Mercurial > public > mercurial-scm > hg-stable

comparison mercurial/pycompat.py @ 44653:00e0c5c06ed5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
pycompat: change argv conversion semantics Use of os.fsencode() to convert Python's sys.argv back to bytes was not correct because it isn't the logically inverse operation from what CPython was doing under the hood. This commit changes the logic for doing the str -> bytes conversion. This required a separate implementation for POSIX and Windows. The Windows behavior is arguably not ideal. The previous behavior on Windows was leading to failing tests, such as test-http-branchmap.t, which defines a utf-8 branch name via a command argument. Previously, Mercurial's argument parser looked to be receiving wchar_t bytes in some cases. After this commit, behavior on Windows is compatible with Python 2, where CPython did not implement `int wmain()` and Windows was performing a Unicode to ANSI conversion on the wchar_t native command line. Arguably better behavior on Windows would be for Mercurial to preserve the original Unicode sequence coming from Python and to wrap this in a bytes-like type so we can round trip safely. But, this would be new, backwards incompatible behavior. My goal for this commit was to converge Mercurial behavior on Python 3 on Windows to fix busted tests. And I believe I was successful, as this commit fixes 9 tests on my Windows machine and 14 tests in the AWS CI environment! Differential Revision: https://phab.mercurial-scm.org/D8337
author Gregory Szorc <gregory.szorc@gmail.com>
date Sat, 28 Mar 2020 12:18:58 -0700
parents 9d2b2df2c2ba
children 7be784f301fa
comparison
equal deleted inserted replaced
44652:949a87145336 44653:00e0c5c06ed5
96 if ispy3: 96 if ispy3:
97 import builtins 97 import builtins
98 import codecs 98 import codecs
99 import functools 99 import functools
100 import io 100 import io
101 import locale
101 import struct 102 import struct
102 103
103 if os.name == r'nt' and sys.version_info >= (3, 6): 104 if os.name == r'nt' and sys.version_info >= (3, 6):
104 # MBCS (or ANSI) filesystem encoding must be used as before. 105 # MBCS (or ANSI) filesystem encoding must be used as before.
105 # Otherwise non-ASCII filenames in existing repositories would be 106 # Otherwise non-ASCII filenames in existing repositories would be
146 # a silly wrapper to make a bytes stream backed by a unicode one. 147 # a silly wrapper to make a bytes stream backed by a unicode one.
147 stdin = sys.stdin.buffer 148 stdin = sys.stdin.buffer
148 stdout = sys.stdout.buffer 149 stdout = sys.stdout.buffer
149 stderr = sys.stderr.buffer 150 stderr = sys.stderr.buffer
150 151
151 # Since Python 3 converts argv to wchar_t type by Py_DecodeLocale() on Unix,
152 # we can use os.fsencode() to get back bytes argv.
153 #
154 # https://hg.python.org/cpython/file/v3.5.1/Programs/python.c#l55
155 #
156 # On Windows, the native argv is unicode and is converted to MBCS bytes
157 # since we do enable the legacy filesystem encoding.
158 if getattr(sys, 'argv', None) is not None: 152 if getattr(sys, 'argv', None) is not None:
159 sysargv = list(map(os.fsencode, sys.argv)) 153 # On POSIX, the char** argv array is converted to Python str using
154 # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
155 # directly callable from Python code. So, we need to emulate it.
156 # Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
157 # surrogateescape error handling on failure. These functions take the
158 # current system locale into account. So, the inverse operation is to
159 # .encode() using the system locale's encoding and using the
160 # surrogateescape error handler. The only tricky part here is getting
161 # the system encoding correct, since `locale.getlocale()` can return
162 # None. We fall back to the filesystem encoding if lookups via `locale`
163 # fail, as this seems like a reasonable thing to do.
164 #
165 # On Windows, the wchar_t **argv is passed into the interpreter as-is.
166 # Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
167 # there's an additional wrinkle. What we really want to access is the
168 # ANSI codepage representation of the arguments, as this is what
169 # `int main()` would receive if Python 3 didn't define `int wmain()`
170 # (this is how Python 2 worked). To get that, we encode with the mbcs
171 # encoding, which will pass CP_ACP to the underlying Windows API to
172 # produce bytes.
173 if os.name == r'nt':
174 sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]
175 else:
176 encoding = (
177 locale.getlocale()[1]
178 or locale.getdefaultlocale()[1]
179 or sys.getfilesystemencoding()
180 )
181 sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]
160 182
161 bytechr = struct.Struct('>B').pack 183 bytechr = struct.Struct('>B').pack
162 byterepr = b'%r'.__mod__ 184 byterepr = b'%r'.__mod__
163 185
164 class bytestr(bytes): 186 class bytestr(bytes):