Is there any way to have output piped line-by-line from a currently executing python program?

When piping printed output from a python script to a command like grep, the output from the script seems to only be piped to the follow-up command after completion of the entire script.

For example, in a script like the following:

#!/usr/bin/env python
from time import sleep

print "message1"
print "message2"
print "message3"

when called with ./ | grep message, nothing will appear for 10 seconds, at which time all three lines will appear.

Compare this to a script

#!/usr/bin/env bash
echo "message1"
sleep 5 
echo "message2"
sleep 5
echo "message3"

./ | grep message will immediately output message1, followed at 5 second intervals by message2 and message3.

I expect this is because only once the python interpreter finishes executing is the output available for the next command. Is there any way to alter this behavior?

Best answer

You can do it:

  • By flushing every print in python
  • By setting stdout to be unbuffered
  • By setting stdout to be line-buffered

You can even call python -u to disable buffering.

I would go for the line-buffering option as it seems most natural.

open(file, mode='r', buffering=-1 ....)

buffering is an optional integer used to set the buffering policy.
Pass 0 to switch buffering off (only allowed in binary mode), 1 to
select line buffering
(only usable in text mode), and an integer > 1
to indicate the size of a fixed-size chunk buffer.

When you don’t specify buffering (the typical “open”) it will use line-buffering if it detects the output is going directly do a TTY, i.e. to your screen console. If you pipe output or redirect it to a file it will switch back to a large (4K / 8K) buffer.

How do you “set stdout to be line-buffered”?

You can reopen stdout via sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1).