Mercurial > public > mercurial-scm > hg

comparison mercurial/utils/procutil.py @ 45095:8e04607023e5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
procutil: ensure that procutil.std{out,err}.write() writes all bytes Python 3 offers different kind of streams and it?s not guaranteed for all of them that calling write() writes all bytes. When Python is started in unbuffered mode, sys.std{out,err}.buffer are instances of io.FileIO, whose write() can write less bytes for platform-specific reasons (e.g. Linux has a 0x7ffff000 bytes maximum and could write less if interrupted by a signal; when writing to Windows consoles, it?s limited to 32767 bytes to avoid the "not enough space" error). This can lead to silent loss of data, both when using sys.std{out,err}.buffer (which may in fact not be a buffered stream) and when using the text streams sys.std{out,err} (I?ve created a CPython bug report for that: https://bugs.python.org/issue41221). Python may fix the problem at some point. For now, we implement our own wrapper for procutil.std{out,err} that calls the raw stream?s write() method until all bytes have been written. We don?t use sys.std{out,err} for larger writes, so I think it?s not worth the effort to patch them.
author Manuel Jacob <me@manueljacob.de>
date Fri, 10 Jul 2020 12:27:58 +0200
parents b4c35e439ea5
children a5fa2761a6cd
comparison
equal deleted inserted replaced
45094:b4c35e439ea5 45095:8e04607023e5
78 if isinstance(stream, LineBufferedWrapper): 78 if isinstance(stream, LineBufferedWrapper):
79 return stream 79 return stream
80 return LineBufferedWrapper(stream) 80 return LineBufferedWrapper(stream)
81 81
82 82
83 class WriteAllWrapper(object):
84 def __init__(self, orig):
85 self.orig = orig
86
87 def __getattr__(self, attr):
88 return getattr(self.orig, attr)
89
90 def write(self, s):
91 write1 = self.orig.write
92 m = memoryview(s)
93 total_to_write = len(s)
94 total_written = 0
95 while total_written < total_to_write:
96 total_written += write1(m[total_written:])
97 return total_written
98
99
100 io.IOBase.register(WriteAllWrapper)
101
102
103 def make_write_all(stream):
104 assert pycompat.ispy3
105 if isinstance(stream, WriteAllWrapper):
106 return stream
107 if isinstance(stream, io.BufferedIOBase):
108 # The io.BufferedIOBase.write() contract guarantees that all data is
109 # written.
110 return stream
111 # In general, the write() method of streams is free to write only part of
112 # the data.
113 return WriteAllWrapper(stream)
114
115
83 if pycompat.ispy3: 116 if pycompat.ispy3:
84 # Python 3 implements its own I/O streams. 117 # Python 3 implements its own I/O streams.
85 # TODO: .buffer might not exist if std streams were replaced; we'll need 118 # TODO: .buffer might not exist if std streams were replaced; we'll need
86 # a silly wrapper to make a bytes stream backed by a unicode one. 119 # a silly wrapper to make a bytes stream backed by a unicode one.
87 stdin = sys.stdin.buffer 120 stdin = sys.stdin.buffer
88 stdout = sys.stdout.buffer 121 stdout = make_write_all(sys.stdout.buffer)
89 if isatty(stdout): 122 if isatty(stdout):
90 # The standard library doesn't offer line-buffered binary streams. 123 # The standard library doesn't offer line-buffered binary streams.
91 stdout = make_line_buffered(stdout) 124 stdout = make_line_buffered(stdout)
92 stderr = sys.stderr.buffer 125 stderr = make_write_all(sys.stderr.buffer)
93 else: 126 else:
94 # Python 2 uses the I/O streams provided by the C library. 127 # Python 2 uses the I/O streams provided by the C library.
95 stdin = sys.stdin 128 stdin = sys.stdin
96 stdout = sys.stdout 129 stdout = sys.stdout
97 if isatty(stdout): 130 if isatty(stdout):