How To Avoid A Broken Pipe Error When Printing A Large Amount Of Formatted Data?

I am trying to print a list of tuples formatted in my stdout. For this, I use the str.format method. Everything works fine, but when I pipe the output to see the first lines using

Solution 1:

head reads from stdout then closes it. This causes print to fail, internally it writes to sys.stdout, now closed.

You can simply catch the IOError and exit silently:

try:
    for pid, uid, pname in data:
        print template.format(pid, uid, pname)
except IOError:
    # stdout is closed, no point in continuing# Attempt to close them explicitly to prevent cleanup problems:try:
        sys.stdout.close()
    except IOError:
        passtry:
        sys.stderr.close()
    except IOError:
        pass

Solution 2:

The behavior you are seeing is linked to the buffered output implementation in Python3. The problem can be avoided using the -u option or setting environmental variable PYTHONUNBUFFERED=x. See the man pages for more information on -u.

$ python2.7 testprint.py | echo

Exc: <type'exceptions.IOError'>
$ python3.5 testprint.py | echo

Exc: <class 'BrokenPipeError'>
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
$ python3.5 -u testprint.py | echo

Exc: <class 'BrokenPipeError'>
$ export PYTHONUNBUFFERED=x
$ python3.5 testprint.py | echo

Exc: <class 'BrokenPipeError'>

Solution 3:

In general, I try to catch the most specific error I can get away with. In this case it is BrokenPipeError:

try:
    # I usually call a function here that generates all my output:for pid, uid, pname in data:
        print template.format(pid, uid, pname)
except BrokenPipeError as e:
    pass# Ignore. Something like head is truncating output.finally:
    sys.stderr.close()

If this is at the end of execution, I find I only need to close sys.stderr. If I don't close sys.stderr, I'll get a BrokenPipeError but without a stack trace.

This seems to be the minimum fix for writing tools that output to pipelines.

Solution 4:

Had this problem with Python3 and debug logging piped into head as well. If your script talks to the network or does file IO, simply dropping IOError's is not a good solution. Despite mentions here, I was not able to catch BrokenPipeError for some reason.

Found a blog post talking about restoring the default signal handler for sigpipe: http://newbebweb.blogspot.com/2012/02/python-head-ioerror-errno-32-broken.html

In short, you add the following to your script before the bulk of the output:

if log.isEnabledFor(logging.DEBUG):  # optional# set default handler to no-opfrom signal import signal, SIGPIPE, SIG_DFL
    signal(SIGPIPE, SIG_DFL)

This seems to happen with head, but not other programs such as grep---as mentioned head closes stdout. If you don't use head with the script often, it may not be worth worrying about.