mercurial-scm/hg: contrib/byteify-strings.py annotate

annotate contrib/byteify-strings.py @ 42674:70bd1965bd07

byteify-strings: handle multi-line strings in _ensuresysstr The current implementation did not handle calls like `repo.ui.log("first line" "other line")` correctly.

author	Rapha?l Gom?s <rgomes@octobus.net>
date	Fri, 02 Aug 2019 09:44:11 +0200
parents	970aaf38c3fc
children	e9592e113c31

rev	line source
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	1 #!/usr/bin/env python3
a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	2 #
38384 1d9c97db465f byteify-strings: fork py3 code transformer to make it a standalone command Yuya Nishihara <yuya@tcha.org> parents: 36617 diff changeset	3 # byteify-strings.py - transform string literals to be Python 3 safe
27220 4374d819ccd5 mercurial: implement import hook for handling C/Python modules Gregory Szorc <gregory.szorc@gmail.com> parents: 0 diff changeset	4 #
4374d819ccd5 mercurial: implement import hook for handling C/Python modules Gregory Szorc <gregory.szorc@gmail.com> parents: 0 diff changeset	5 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
4374d819ccd5 mercurial: implement import hook for handling C/Python modules Gregory Szorc <gregory.szorc@gmail.com> parents: 0 diff changeset	6 #
4374d819ccd5 mercurial: implement import hook for handling C/Python modules Gregory Szorc <gregory.szorc@gmail.com> parents: 0 diff changeset	7 # This software may be used and distributed according to the terms of the
4374d819ccd5 mercurial: implement import hook for handling C/Python modules Gregory Szorc <gregory.szorc@gmail.com> parents: 0 diff changeset	8 # GNU General Public License version 2 or any later version.
4374d819ccd5 mercurial: implement import hook for handling C/Python modules Gregory Szorc <gregory.szorc@gmail.com> parents: 0 diff changeset	9
42247 970aaf38c3fc contrib: have byteify-strings explode if run in Python 2 Augie Fackler <augie@google.com> parents: 39103 diff changeset	10 from __future__ import absolute_import, print_function
27220 4374d819ccd5 mercurial: implement import hook for handling C/Python modules Gregory Szorc <gregory.szorc@gmail.com> parents: 0 diff changeset	11
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	12 import argparse
38386 9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	13 import contextlib
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	14 import errno
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	15 import os
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	16 import sys
38386 9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	17 import tempfile
38384 1d9c97db465f byteify-strings: fork py3 code transformer to make it a standalone command Yuya Nishihara <yuya@tcha.org> parents: 36617 diff changeset	18 import token
1d9c97db465f byteify-strings: fork py3 code transformer to make it a standalone command Yuya Nishihara <yuya@tcha.org> parents: 36617 diff changeset	19 import tokenize
29550 1c22400db72d mercurial: implement a source transforming module loader on Python 3 Gregory Szorc <gregory.szorc@gmail.com> parents: 29490 diff changeset	20
38390 47dd23e6b116 byteify-strings: try to preserve column alignment Yuya Nishihara <yuya@tcha.org> parents: 38389 diff changeset	21 def adjusttokenpos(t, ofs):
47dd23e6b116 byteify-strings: try to preserve column alignment Yuya Nishihara <yuya@tcha.org> parents: 38389 diff changeset	22 """Adjust start/end column of the given token"""
47dd23e6b116 byteify-strings: try to preserve column alignment Yuya Nishihara <yuya@tcha.org> parents: 38389 diff changeset	23 return t._replace(start=(t.start[0], t.start[1] + ofs),
47dd23e6b116 byteify-strings: try to preserve column alignment Yuya Nishihara <yuya@tcha.org> parents: 38389 diff changeset	24 end=(t.end[0], t.end[1] + ofs))
47dd23e6b116 byteify-strings: try to preserve column alignment Yuya Nishihara <yuya@tcha.org> parents: 38389 diff changeset	25
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	26 def replacetokens(tokens, opts):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	27 """Transform a stream of tokens from raw to Python 3.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	28
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	29 Returns a generator of possibly rewritten tokens.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	30
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	31 The input token list may be mutated as part of processing. However,
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	32 its changes do not necessarily match the output token stream.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	33 """
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	34 sysstrtokens = set()
29550 1c22400db72d mercurial: implement a source transforming module loader on Python 3 Gregory Szorc <gregory.szorc@gmail.com> parents: 29490 diff changeset	35
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	36 # The following utility functions access the tokens list and i index of
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	37 # the for i, t enumerate(tokens) loop below
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	38 def _isop(j, *o):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	39 """Assert that tokens[j] is an OP with one of the given values"""
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	40 try:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	41 return tokens[j].type == token.OP and tokens[j].string in o
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	42 except IndexError:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	43 return False
29550 1c22400db72d mercurial: implement a source transforming module loader on Python 3 Gregory Szorc <gregory.szorc@gmail.com> parents: 29490 diff changeset	44
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	45 def _findargnofcall(n):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	46 """Find arg n of a call expression (start at 0)
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	47
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	48 Returns index of the first token of that argument, or None if
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	49 there is not that many arguments.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	50
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	51 Assumes that token[i + 1] is '('.
38389 1d68fd5f614a byteify-strings: do not rewrite system string literals to u'' Yuya Nishihara <yuya@tcha.org> parents: 38388 diff changeset	52
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	53 """
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	54 nested = 0
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	55 for j in range(i + 2, len(tokens)):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	56 if _isop(j, ')', ']', '}'):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	57 # end of call, tuple, subscription or dict / set
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	58 nested -= 1
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	59 if nested < 0:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	60 return None
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	61 elif n == 0:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	62 # this is the starting position of arg
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	63 return j
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	64 elif _isop(j, '(', '[', '{'):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	65 nested += 1
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	66 elif _isop(j, ',') and nested == 0:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	67 n -= 1
30165 423377290a3a py3: refactor token parsing to handle call args properly Martijn Pieters <mjpieters@fb.com> parents: 30118 diff changeset	68
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	69 return None
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	70
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	71 def _ensuresysstr(j):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	72 """Make sure the token at j is a system string
30165 423377290a3a py3: refactor token parsing to handle call args properly Martijn Pieters <mjpieters@fb.com> parents: 30118 diff changeset	73
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	74 Remember the given token so the string transformer won't add
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	75 the byte prefix.
30165 423377290a3a py3: refactor token parsing to handle call args properly Martijn Pieters <mjpieters@fb.com> parents: 30118 diff changeset	76
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	77 Ignores tokens that are not strings. Assumes bounds checking has
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	78 already been done.
30165 423377290a3a py3: refactor token parsing to handle call args properly Martijn Pieters <mjpieters@fb.com> parents: 30118 diff changeset	79
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	80 """
42674 70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	81 k = j
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	82 currtoken = tokens[k]
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	83 while currtoken.type in (token.STRING, token.NEWLINE, tokenize.NL):
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	84 k += 1
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	85 if (
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	86 currtoken.type == token.STRING
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	87 and currtoken.string.startswith(("'", '"'))
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	88 ):
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	89 sysstrtokens.add(currtoken)
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	90 try:
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	91 currtoken = tokens[k]
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	92 except IndexError:
70bd1965bd07 byteify-strings: handle multi-line strings in _ensuresysstr Rapha?l Gom?s <rgomes@octobus.net> parents: 42247 diff changeset	93 break
30165 423377290a3a py3: refactor token parsing to handle call args properly Martijn Pieters <mjpieters@fb.com> parents: 30118 diff changeset	94
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	95 coldelta = 0 # column increment for new opening parens
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	96 coloffset = -1 # column offset for the current line (-1: TBD)
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	97 parens = [(0, 0, 0)] # stack of (line, end-column, column-offset)
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	98 for i, t in enumerate(tokens):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	99 # Compute the column offset for the current line, such that
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	100 # the current line will be aligned to the last opening paren
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	101 # as before.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	102 if coloffset < 0:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	103 if t.start[1] == parens[-1][1]:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	104 coloffset = parens[-1][2]
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	105 elif t.start[1] + 1 == parens[-1][1]:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	106 # fix misaligned indent of s/util.Abort/error.Abort/
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	107 coloffset = parens[-1][2] + (parens[-1][1] - t.start[1])
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	108 else:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	109 coloffset = 0
30165 423377290a3a py3: refactor token parsing to handle call args properly Martijn Pieters <mjpieters@fb.com> parents: 30118 diff changeset	110
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	111 # Reset per-line attributes at EOL.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	112 if t.type in (token.NEWLINE, tokenize.NL):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	113 yield adjusttokenpos(t, coloffset)
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	114 coldelta = 0
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	115 coloffset = -1
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	116 continue
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	117
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	118 # Remember the last paren position.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	119 if _isop(i, '(', '[', '{'):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	120 parens.append(t.end + (coloffset + coldelta,))
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	121 elif _isop(i, ')', ']', '}'):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	122 parens.pop()
30165 423377290a3a py3: refactor token parsing to handle call args properly Martijn Pieters <mjpieters@fb.com> parents: 30118 diff changeset	123
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	124 # Convert most string literals to byte literals. String literals
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	125 # in Python 2 are bytes. String literals in Python 3 are unicode.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	126 # Most strings in Mercurial are bytes and unicode strings are rare.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	127 # Rather than rewrite all string literals to use ``b''`` to indicate
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	128 # byte strings, we apply this token transformer to insert the ``b``
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	129 # prefix nearly everywhere.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	130 if t.type == token.STRING and t not in sysstrtokens:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	131 s = t.string
38390 47dd23e6b116 byteify-strings: try to preserve column alignment Yuya Nishihara <yuya@tcha.org> parents: 38389 diff changeset	132
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	133 # Preserve docstrings as string literals. This is inconsistent
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	134 # with regular unprefixed strings. However, the
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	135 # "from __future__" parsing (which allows a module docstring to
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	136 # exist before it) doesn't properly handle the docstring if it
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	137 # is b''' prefixed, leading to a SyntaxError. We leave all
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	138 # docstrings as unprefixed to avoid this. This means Mercurial
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	139 # components touching docstrings need to handle unicode,
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	140 # unfortunately.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	141 if s[0:3] in ("'''", '"""'):
38390 47dd23e6b116 byteify-strings: try to preserve column alignment Yuya Nishihara <yuya@tcha.org> parents: 38389 diff changeset	142 yield adjusttokenpos(t, coloffset)
47dd23e6b116 byteify-strings: try to preserve column alignment Yuya Nishihara <yuya@tcha.org> parents: 38389 diff changeset	143 continue
47dd23e6b116 byteify-strings: try to preserve column alignment Yuya Nishihara <yuya@tcha.org> parents: 38389 diff changeset	144
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	145 # If the first character isn't a quote, it is likely a string
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	146 # prefixing character (such as 'b', 'u', or 'r'. Ignore.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	147 if s[0] not in ("'", '"'):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	148 yield adjusttokenpos(t, coloffset)
29550 1c22400db72d mercurial: implement a source transforming module loader on Python 3 Gregory Szorc <gregory.szorc@gmail.com> parents: 29490 diff changeset	149 continue
1c22400db72d mercurial: implement a source transforming module loader on Python 3 Gregory Szorc <gregory.szorc@gmail.com> parents: 29490 diff changeset	150
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	151 # String literal. Prefix to make a b'' string.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	152 yield adjusttokenpos(t._replace(string='b%s' % t.string),
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	153 coloffset)
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	154 coldelta += 1
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	155 continue
29550 1c22400db72d mercurial: implement a source transforming module loader on Python 3 Gregory Szorc <gregory.szorc@gmail.com> parents: 29490 diff changeset	156
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	157 # This looks like a function call.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	158 if t.type == token.NAME and _isop(i + 1, '('):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	159 fn = t.string
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	160
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	161 # *attr() builtins don't accept byte strings to 2nd argument.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	162 if (fn in ('getattr', 'setattr', 'hasattr', 'safehasattr') and
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	163 not _isop(i - 1, '.')):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	164 arg1idx = _findargnofcall(1)
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	165 if arg1idx is not None:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	166 _ensuresysstr(arg1idx)
29550 1c22400db72d mercurial: implement a source transforming module loader on Python 3 Gregory Szorc <gregory.szorc@gmail.com> parents: 29490 diff changeset	167
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	168 # .encode() and .decode() on str/bytes/unicode don't accept
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	169 # byte strings on Python 3.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	170 elif fn in ('encode', 'decode') and _isop(i - 1, '.'):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	171 for argn in range(2):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	172 argidx = _findargnofcall(argn)
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	173 if argidx is not None:
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	174 _ensuresysstr(argidx)
29550 1c22400db72d mercurial: implement a source transforming module loader on Python 3 Gregory Szorc <gregory.szorc@gmail.com> parents: 29490 diff changeset	175
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	176 # It changes iteritems/values to items/values as they are not
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	177 # present in Python 3 world.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	178 elif opts['dictiter'] and fn in ('iteritems', 'itervalues'):
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	179 yield adjusttokenpos(t._replace(string=fn[4:]), coloffset)
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	180 continue
30052 eaaedad68011 py3: switch to .items() using transformer Pulkit Goyal <7895pulkit@gmail.com> parents: 30051 diff changeset	181
39103 da130c5cef90 byteify-strings: prevent "__name__ == '__main__'" from being transformed Yuya Nishihara <yuya@tcha.org> parents: 38391 diff changeset	182 # Looks like "if __name__ == '__main__'".
da130c5cef90 byteify-strings: prevent "__name__ == '__main__'" from being transformed Yuya Nishihara <yuya@tcha.org> parents: 38391 diff changeset	183 if (t.type == token.NAME and t.string == '__name__'
da130c5cef90 byteify-strings: prevent "__name__ == '__main__'" from being transformed Yuya Nishihara <yuya@tcha.org> parents: 38391 diff changeset	184 and _isop(i + 1, '==')):
da130c5cef90 byteify-strings: prevent "__name__ == '__main__'" from being transformed Yuya Nishihara <yuya@tcha.org> parents: 38391 diff changeset	185 _ensuresysstr(i + 2)
da130c5cef90 byteify-strings: prevent "__name__ == '__main__'" from being transformed Yuya Nishihara <yuya@tcha.org> parents: 38391 diff changeset	186
38391 f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	187 # Emit unmodified token.
f77bbd34a1df byteify-strings: remove superfluous "if True" block Yuya Nishihara <yuya@tcha.org> parents: 38390 diff changeset	188 yield adjusttokenpos(t, coloffset)
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	189
38388 f701bc936e7f byteify-strings: do not rewrite iteritems() and itervalues() by default Yuya Nishihara <yuya@tcha.org> parents: 38387 diff changeset	190 def process(fin, fout, opts):
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	191 tokens = tokenize.tokenize(fin.readline)
38388 f701bc936e7f byteify-strings: do not rewrite iteritems() and itervalues() by default Yuya Nishihara <yuya@tcha.org> parents: 38387 diff changeset	192 tokens = replacetokens(list(tokens), opts)
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	193 fout.write(tokenize.untokenize(tokens))
a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	194
38386 9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	195 def tryunlink(fname):
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	196 try:
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	197 os.unlink(fname)
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	198 except OSError as err:
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	199 if err.errno != errno.ENOENT:
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	200 raise
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	201
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	202 @contextlib.contextmanager
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	203 def editinplace(fname):
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	204 n = os.path.basename(fname)
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	205 d = os.path.dirname(fname)
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	206 fp = tempfile.NamedTemporaryFile(prefix='.%s-' % n, suffix='~', dir=d,
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	207 delete=False)
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	208 try:
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	209 yield fp
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	210 fp.close()
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	211 if os.name == 'nt':
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	212 tryunlink(fname)
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	213 os.rename(fp.name, fname)
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	214 finally:
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	215 fp.close()
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	216 tryunlink(fp.name)
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	217
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	218 def main():
a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	219 ap = argparse.ArgumentParser()
38386 9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	220 ap.add_argument('-i', '--inplace', action='store_true', default=False,
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	221 help='edit files in place')
38388 f701bc936e7f byteify-strings: do not rewrite iteritems() and itervalues() by default Yuya Nishihara <yuya@tcha.org> parents: 38387 diff changeset	222 ap.add_argument('--dictiter', action='store_true', default=False,
f701bc936e7f byteify-strings: do not rewrite iteritems() and itervalues() by default Yuya Nishihara <yuya@tcha.org> parents: 38387 diff changeset	223 help='rewrite iteritems() and itervalues()'),
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	224 ap.add_argument('files', metavar='FILE', nargs='+', help='source file')
a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	225 args = ap.parse_args()
38388 f701bc936e7f byteify-strings: do not rewrite iteritems() and itervalues() by default Yuya Nishihara <yuya@tcha.org> parents: 38387 diff changeset	226 opts = {
f701bc936e7f byteify-strings: do not rewrite iteritems() and itervalues() by default Yuya Nishihara <yuya@tcha.org> parents: 38387 diff changeset	227 'dictiter': args.dictiter,
f701bc936e7f byteify-strings: do not rewrite iteritems() and itervalues() by default Yuya Nishihara <yuya@tcha.org> parents: 38387 diff changeset	228 }
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	229 for fname in args.files:
38386 9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	230 if args.inplace:
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	231 with editinplace(fname) as fout:
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	232 with open(fname, 'rb') as fin:
38388 f701bc936e7f byteify-strings: do not rewrite iteritems() and itervalues() by default Yuya Nishihara <yuya@tcha.org> parents: 38387 diff changeset	233 process(fin, fout, opts)
38386 9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	234 else:
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	235 with open(fname, 'rb') as fin:
9f42e4a83676 byteify-strings: add --inplace option to write back result Yuya Nishihara <yuya@tcha.org> parents: 38385 diff changeset	236 fout = sys.stdout.buffer
38388 f701bc936e7f byteify-strings: do not rewrite iteritems() and itervalues() by default Yuya Nishihara <yuya@tcha.org> parents: 38387 diff changeset	237 process(fin, fout, opts)
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	238
a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	239 if __name__ == '__main__':
42247 970aaf38c3fc contrib: have byteify-strings explode if run in Python 2 Augie Fackler <augie@google.com> parents: 39103 diff changeset	240 if sys.version_info.major < 3:
970aaf38c3fc contrib: have byteify-strings explode if run in Python 2 Augie Fackler <augie@google.com> parents: 39103 diff changeset	241 print('This script must be run under Python 3.')
970aaf38c3fc contrib: have byteify-strings explode if run in Python 2 Augie Fackler <augie@google.com> parents: 39103 diff changeset	242 sys.exit(3)
38385 a2976c27dac4 byteify-strings: add basic command interface Yuya Nishihara <yuya@tcha.org> parents: 38384 diff changeset	243 main()

Mercurial > public > mercurial-scm > hg

annotate contrib/byteify-strings.py @ 42674:70bd1965bd07