When piping printed output from a python script to a command like grep, the output from the script seems to only be piped to the follow-up command after completion of the entire script.
For example, in a script
test_grep.py like the following:
#!/usr/bin/env python from time import sleep print "message1" sleep(5) print "message2" sleep(5) print "message3"
when called with
./test_grep.py | grep message, nothing will appear for 10 seconds, at which time all three lines will appear.
Compare this to a script
#!/usr/bin/env bash echo "message1" sleep 5 echo "message2" sleep 5 echo "message3"
./test_grep.sh | grep message will immediately output
message1, followed at 5 second intervals by
I expect this is because only once the python interpreter finishes executing is the output available for the next command. Is there any way to alter this behavior?
You can do it:
- By flushing every
- By setting stdout to be unbuffered
- By setting stdout to be line-buffered
You can even call
python -u to disable buffering.
I would go for the line-buffering option as it seems most natural.
open(file, mode='r', buffering=-1 ....)
buffering is an optional integer used to set the buffering policy.
Pass 0 to switch buffering off (only allowed in binary mode), 1 to
select line buffering (only usable in text mode), and an integer > 1
to indicate the size of a fixed-size chunk buffer.
When you don’t specify buffering (the typical “open”) it will use line-buffering if it detects the output is going directly do a TTY, i.e. to your screen console. If you pipe output or redirect it to a file it will switch back to a large (4K / 8K) buffer.
How do you “set stdout to be line-buffered”?
You can reopen
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1).