mercurial-scm/hg-stable: mercurial/encoding.py annotate

annotate mercurial/encoding.py @ 30033:02dbfaa6df0b

py3: convert encoding name and mode to str Otherwise tolocal() and fromlocal() wouldn't work on Python 3. Still tolocal() can't make a valid localstr object because localstr inherits str, but it can return some object without raising exceptions. Since Py3 bytes() behaves much like bytearray() than str() of Py2, we can't simply do s/str/bytes/g. I have no good idea to handle str/bytes divergence.

author	Yuya Nishihara <yuya@tcha.org>
date	Wed, 28 Sep 2016 20:39:06 +0900
parents	0f6d6fdd3c2a
children	e4a6b439acc5

rev	line source
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	1 # encoding.py - character transcoding support for Mercurial
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	3 # Copyright 2005-2009 Matt Mackall <mpm@selenic.com> and others
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	4 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 9574 diff changeset	6 # GNU General Public License version 2 or any later version.
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	7
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	8 from __future__ import absolute_import
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	9
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	10 import array
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	11 import locale
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	12 import os
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	13 import unicodedata
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	14
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	15 from . import (
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	16 error,
30031 0f6d6fdd3c2a pycompat: provide 'ispy3' constant Yuya Nishihara <yuya@tcha.org> parents: 28508 diff changeset	17 pycompat,
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	18 )
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	19
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	20 _sysstr = pycompat.sysstr
02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	21
30031 0f6d6fdd3c2a pycompat: provide 'ispy3' constant Yuya Nishihara <yuya@tcha.org> parents: 28508 diff changeset	22 if pycompat.ispy3:
28507 9bcbd9412225 encoding: make HFS+ ignore code Python 3 compatible Gregory Szorc <gregory.szorc@gmail.com> parents: 28069 diff changeset	23 unichr = chr
9bcbd9412225 encoding: make HFS+ ignore code Python 3 compatible Gregory Szorc <gregory.szorc@gmail.com> parents: 28069 diff changeset	24
23596 885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	25 # These unicode characters are ignored by HFS+ (Apple Technote 1150,
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	26 # "Unicode Subtleties"), so we need to ignore them in some places for
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	27 # sanity.
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	28 _ignore = [unichr(int(x, 16)).encode("utf-8") for x in
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	29 "200c 200d 200e 200f 202a 202b 202c 202d 202e "
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	30 "206a 206b 206c 206d 206e 206f feff".split()]
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	31 # verify the next function will work
30031 0f6d6fdd3c2a pycompat: provide 'ispy3' constant Yuya Nishihara <yuya@tcha.org> parents: 28508 diff changeset	32 if pycompat.ispy3:
28507 9bcbd9412225 encoding: make HFS+ ignore code Python 3 compatible Gregory Szorc <gregory.szorc@gmail.com> parents: 28069 diff changeset	33 assert set(i[0] for i in _ignore) == set([ord(b'\xe2'), ord(b'\xef')])
9bcbd9412225 encoding: make HFS+ ignore code Python 3 compatible Gregory Szorc <gregory.szorc@gmail.com> parents: 28069 diff changeset	34 else:
9bcbd9412225 encoding: make HFS+ ignore code Python 3 compatible Gregory Szorc <gregory.szorc@gmail.com> parents: 28069 diff changeset	35 assert set(i[0] for i in _ignore) == set(["\xe2", "\xef"])
23596 885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	36
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	37 def hfsignoreclean(s):
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	38 """Remove codepoints ignored by HFS+ from s.
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	39
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	40 >>> hfsignoreclean(u'.h\u200cg'.encode('utf-8'))
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	41 '.hg'
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	42 >>> hfsignoreclean(u'.h\ufeffg'.encode('utf-8'))
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	43 '.hg'
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	44 """
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	45 if "\xe2" in s or "\xef" in s:
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	46 for c in _ignore:
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	47 s = s.replace(c, '')
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	48 return s
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	49
11892 2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	50 def _getpreferredencoding():
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	51 '''
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	52 On darwin, getpreferredencoding ignores the locale environment and
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	53 always returns mac-roman. http://bugs.python.org/issue6202 fixes this
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	54 for Python 2.7 and up. This is the same corrected code for earlier
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	55 Python versions.
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	56
12770 614f0d8724ab check-code: find trailing whitespace Martin Geisler <mg@lazybytes.net> parents: 12062 diff changeset	57 However, we can't use a version check for this method, as some distributions
11892 2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	58 patch Python to fix this. Instead, we use it as a 'fixer' for the mac-roman
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	59 encoding, as it is unlikely that this encoding is the actually expected.
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	60 '''
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	61 try:
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	62 locale.CODESET
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	63 except AttributeError:
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	64 # Fall back to parsing environment variables :-(
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	65 return locale.getdefaultlocale()[1]
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	66
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	67 oldloc = locale.setlocale(locale.LC_CTYPE)
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	68 locale.setlocale(locale.LC_CTYPE, "")
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	69 result = locale.nl_langinfo(locale.CODESET)
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	70 locale.setlocale(locale.LC_CTYPE, oldloc)
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	71
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	72 return result
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	73
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	74 _encodingfixers = {
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	75 '646': lambda: 'ascii',
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	76 'ANSI_X3.4-1968': lambda: 'ascii',
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	77 'mac-roman': _getpreferredencoding
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	78 }
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	79
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	80 try:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	81 encoding = os.environ.get("HGENCODING")
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	82 if not encoding:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	83 encoding = locale.getpreferredencoding() or 'ascii'
11892 2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	84 encoding = _encodingfixers.get(encoding, lambda: encoding)()
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	85 except locale.Error:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	86 encoding = 'ascii'
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	87 encodingmode = os.environ.get("HGENCODINGMODE", "strict")
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	88 fallbackencoding = 'ISO-8859-1'
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	89
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	90 class localstr(str):
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	91 '''This class allows strings that are unmodified to be
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	92 round-tripped to the local encoding and back'''
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	93 def __new__(cls, u, l):
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	94 s = str.__new__(cls, l)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	95 s._utf8 = u
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	96 return s
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	97 def __hash__(self):
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	98 return hash(self._utf8) # avoid collisions in local string space
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	99
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	100 def tolocal(s):
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	101 """
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	102 Convert a string from internal UTF-8 to local encoding
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	103
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	104 All internal strings should be UTF-8 but some repos before the
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	105 implementation of locale support may contain latin1 or possibly
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	106 other character sets. We attempt to decode everything strictly
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	107 using UTF-8, then Latin-1, and failing that, we use UTF-8 and
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	108 replace unknown characters.
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	109
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	110 The localstr class is used to cache the known UTF-8 encoding of
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	111 strings next to their local representation to allow lossless
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	112 round-trip conversion back to UTF-8.
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	113
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	114 >>> u = 'foo: \\xc3\\xa4' # utf-8
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	115 >>> l = tolocal(u)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	116 >>> l
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	117 'foo: ?'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	118 >>> fromlocal(l)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	119 'foo: \\xc3\\xa4'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	120 >>> u2 = 'foo: \\xc3\\xa1'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	121 >>> d = { l: 1, tolocal(u2): 2 }
18378 404feac78b8a tests: stabilize doctest output Mads Kiilerich <mads@kiilerich.com> parents: 17424 diff changeset	122 >>> len(d) # no collision
404feac78b8a tests: stabilize doctest output Mads Kiilerich <mads@kiilerich.com> parents: 17424 diff changeset	123 2
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	124 >>> 'foo: ?' in d
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	125 False
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	126 >>> l1 = 'foo: \\xe4' # historical latin1 fallback
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	127 >>> l = tolocal(l1)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	128 >>> l
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	129 'foo: ?'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	130 >>> fromlocal(l) # magically in utf-8
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	131 'foo: \\xc3\\xa4'
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	132 """
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	133
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	134 try:
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	135 try:
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	136 # make sure string is actually stored in UTF-8
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	137 u = s.decode('UTF-8')
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	138 if encoding == 'UTF-8':
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	139 # fast path
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	140 return s
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	141 r = u.encode(_sysstr(encoding), u"replace")
02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	142 if u == r.decode(_sysstr(encoding)):
13940 b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	143 # r is a safe, non-lossy encoding of s
b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	144 return r
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	145 return localstr(s, r)
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	146 except UnicodeDecodeError:
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	147 # we should only get here if we're looking at an ancient changeset
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	148 try:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	149 u = s.decode(_sysstr(fallbackencoding))
02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	150 r = u.encode(_sysstr(encoding), u"replace")
02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	151 if u == r.decode(_sysstr(encoding)):
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	152 # r is a safe, non-lossy encoding of s
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	153 return r
13940 b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	154 return localstr(u.encode('UTF-8'), r)
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	155 except UnicodeDecodeError:
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	156 u = s.decode("utf-8", "replace") # last ditch
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	157 # can't round-trip
02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	158 return u.encode(_sysstr(encoding), u"replace")
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	159 except LookupError as k:
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	160 raise error.Abort(k, hint="please check your locale settings")
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	161
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	162 def fromlocal(s):
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	163 """
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	164 Convert a string from the local character encoding to UTF-8
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	165
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	166 We attempt to decode strings using the encoding mode set by
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	167 HGENCODINGMODE, which defaults to 'strict'. In this mode, unknown
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	168 characters will cause an error message. Other modes include
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	169 'replace', which replaces unknown characters with a special
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	170 Unicode character, and 'ignore', which drops the character.
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	171 """
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	172
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	173 # can we do a lossless round-trip?
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	174 if isinstance(s, localstr):
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	175 return s._utf8
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	176
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	177 try:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	178 u = s.decode(_sysstr(encoding), _sysstr(encodingmode))
02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	179 return u.encode("utf-8")
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	180 except UnicodeDecodeError as inst:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10263 diff changeset	181 sub = s[max(0, inst.start - 10):inst.start + 10]
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	182 raise error.Abort("decoding near '%s': %s!" % (sub, inst))
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	183 except LookupError as k:
15769 afdf4f5bac61 encoding: use hint markup for "please check your locale settings" Mads Kiilerich <mads@kiilerich.com> parents: 15672 diff changeset	184 raise error.Abort(k, hint="please check your locale settings")
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	185
12866 eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	186 # How to treat ambiguous-width characters. Set to 'wide' to treat as wide.
15066 24efa83d81cb i18n: calculate terminal columns by width information of each characters FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 14951 diff changeset	187 wide = (os.environ.get("HGENCODINGAMBIGUOUS", "narrow") == "wide"
24efa83d81cb i18n: calculate terminal columns by width information of each characters FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 14951 diff changeset	188 and "WFA" or "WF")
12866 eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	189
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	190 def colwidth(s):
15142 176882876780 encoding: colwidth input is in the local encoding Matt Mackall <mpm@selenic.com> parents: 15066 diff changeset	191 "Find the column width of a string for display in the local encoding"
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	192 return ucolwidth(s.decode(_sysstr(encoding), u'replace'))
15066 24efa83d81cb i18n: calculate terminal columns by width information of each characters FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 14951 diff changeset	193
24efa83d81cb i18n: calculate terminal columns by width information of each characters FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 14951 diff changeset	194 def ucolwidth(d):
24efa83d81cb i18n: calculate terminal columns by width information of each characters FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 14951 diff changeset	195 "Find the column width of a Unicode string for display"
14951 61807854004e encoding: use getattr isntead of hasattr Augie Fackler <durin42@gmail.com> parents: 14069 diff changeset	196 eaw = getattr(unicodedata, 'east_asian_width', None)
61807854004e encoding: use getattr isntead of hasattr Augie Fackler <durin42@gmail.com> parents: 14069 diff changeset	197 if eaw is not None:
61807854004e encoding: use getattr isntead of hasattr Augie Fackler <durin42@gmail.com> parents: 14069 diff changeset	198 return sum([eaw(c) in wide and 2 or 1 for c in d])
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	199 return len(d)
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	200
15143 16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	201 def getcols(s, start, c):
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	202 '''Use colwidth to find a c-column substring of s starting at byte
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	203 index start'''
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	204 for x in xrange(start + c, len(s)):
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	205 t = s[start:x]
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	206 if colwidth(t) == c:
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	207 return t
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	208
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	209 def trim(s, width, ellipsis='', leftside=False):
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	210 """Trim string 's' to at most 'width' columns (including 'ellipsis').
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	211
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	212 If 'leftside' is True, left side of string 's' is trimmed.
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	213 'ellipsis' is always placed at trimmed side.
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	214
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	215 >>> ellipsis = '+++'
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	216 >>> from . import encoding
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	217 >>> encoding.encoding = 'utf-8'
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	218 >>> t= '1234567890'
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	219 >>> print trim(t, 12, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	220 1234567890
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	221 >>> print trim(t, 10, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	222 1234567890
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	223 >>> print trim(t, 8, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	224 12345+++
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	225 >>> print trim(t, 8, ellipsis=ellipsis, leftside=True)
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	226 +++67890
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	227 >>> print trim(t, 8)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	228 12345678
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	229 >>> print trim(t, 8, leftside=True)
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	230 34567890
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	231 >>> print trim(t, 3, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	232 +++
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	233 >>> print trim(t, 1, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	234 +
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	235 >>> u = u'\u3042\u3044\u3046\u3048\u304a' # 2 x 5 = 10 columns
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	236 >>> t = u.encode(encoding.encoding)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	237 >>> print trim(t, 12, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	238 \xe3\x81\x82\xe3\x81\x84\xe3\x81\x86\xe3\x81\x88\xe3\x81\x8a
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	239 >>> print trim(t, 10, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	240 \xe3\x81\x82\xe3\x81\x84\xe3\x81\x86\xe3\x81\x88\xe3\x81\x8a
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	241 >>> print trim(t, 8, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	242 \xe3\x81\x82\xe3\x81\x84+++
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	243 >>> print trim(t, 8, ellipsis=ellipsis, leftside=True)
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	244 +++\xe3\x81\x88\xe3\x81\x8a
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	245 >>> print trim(t, 5)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	246 \xe3\x81\x82\xe3\x81\x84
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	247 >>> print trim(t, 5, leftside=True)
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	248 \xe3\x81\x88\xe3\x81\x8a
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	249 >>> print trim(t, 4, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	250 +++
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	251 >>> print trim(t, 4, ellipsis=ellipsis, leftside=True)
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	252 +++
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	253 >>> t = '\x11\x22\x33\x44\x55\x66\x77\x88\x99\xaa' # invalid byte sequence
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	254 >>> print trim(t, 12, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	255 \x11\x22\x33\x44\x55\x66\x77\x88\x99\xaa
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	256 >>> print trim(t, 10, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	257 \x11\x22\x33\x44\x55\x66\x77\x88\x99\xaa
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	258 >>> print trim(t, 8, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	259 \x11\x22\x33\x44\x55+++
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	260 >>> print trim(t, 8, ellipsis=ellipsis, leftside=True)
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	261 +++\x66\x77\x88\x99\xaa
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	262 >>> print trim(t, 8)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	263 \x11\x22\x33\x44\x55\x66\x77\x88
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	264 >>> print trim(t, 8, leftside=True)
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	265 \x33\x44\x55\x66\x77\x88\x99\xaa
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	266 >>> print trim(t, 3, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	267 +++
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	268 >>> print trim(t, 1, ellipsis=ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	269 +
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	270 """
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	271 try:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	272 u = s.decode(_sysstr(encoding))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	273 except UnicodeDecodeError:
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	274 if len(s) <= width: # trimming is not needed
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	275 return s
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	276 width -= len(ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	277 if width <= 0: # no enough room even for ellipsis
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	278 return ellipsis[:width + len(ellipsis)]
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	279 if leftside:
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	280 return ellipsis + s[-width:]
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	281 return s[:width] + ellipsis
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	282
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	283 if ucolwidth(u) <= width: # trimming is not needed
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	284 return s
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	285
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	286 width -= len(ellipsis)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	287 if width <= 0: # no enough room even for ellipsis
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	288 return ellipsis[:width + len(ellipsis)]
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	289
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	290 if leftside:
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	291 uslice = lambda i: u[i:]
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	292 concat = lambda s: ellipsis + s
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	293 else:
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	294 uslice = lambda i: u[:-i]
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	295 concat = lambda s: s + ellipsis
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	296 for i in xrange(1, len(u)):
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	297 usub = uslice(i)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	298 if ucolwidth(usub) <= width:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	299 return concat(usub.encode(_sysstr(encoding)))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	300 return ellipsis # no enough room for multi-column characters
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	301
22973 bcff9ecdaae0 encoding: avoid cyclic dependency around "parsers" in pure Python build FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22779 diff changeset	302 def _asciilower(s):
22778 80f2b63dd83a parsers: add a function to efficiently lowercase ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 22426 diff changeset	303 '''convert a string to lowercase if ASCII
80f2b63dd83a parsers: add a function to efficiently lowercase ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 22426 diff changeset	304
80f2b63dd83a parsers: add a function to efficiently lowercase ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 22426 diff changeset	305 Raises UnicodeDecodeError if non-ASCII characters are found.'''
80f2b63dd83a parsers: add a function to efficiently lowercase ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 22426 diff changeset	306 s.decode('ascii')
80f2b63dd83a parsers: add a function to efficiently lowercase ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 22426 diff changeset	307 return s.lower()
80f2b63dd83a parsers: add a function to efficiently lowercase ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 22426 diff changeset	308
22973 bcff9ecdaae0 encoding: avoid cyclic dependency around "parsers" in pure Python build FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22779 diff changeset	309 def asciilower(s):
bcff9ecdaae0 encoding: avoid cyclic dependency around "parsers" in pure Python build FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22779 diff changeset	310 # delay importing avoids cyclic dependency around "parsers" in
bcff9ecdaae0 encoding: avoid cyclic dependency around "parsers" in pure Python build FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22779 diff changeset	311 # pure Python build (util => i18n => encoding => parsers => util)
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	312 from . import parsers
22973 bcff9ecdaae0 encoding: avoid cyclic dependency around "parsers" in pure Python build FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22779 diff changeset	313 impl = getattr(parsers, 'asciilower', _asciilower)
bcff9ecdaae0 encoding: avoid cyclic dependency around "parsers" in pure Python build FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22779 diff changeset	314 global asciilower
bcff9ecdaae0 encoding: avoid cyclic dependency around "parsers" in pure Python build FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22779 diff changeset	315 asciilower = impl
bcff9ecdaae0 encoding: avoid cyclic dependency around "parsers" in pure Python build FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 22779 diff changeset	316 return impl(s)
22778 80f2b63dd83a parsers: add a function to efficiently lowercase ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 22426 diff changeset	317
24578 ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	318 def _asciiupper(s):
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	319 '''convert a string to uppercase if ASCII
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	320
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	321 Raises UnicodeDecodeError if non-ASCII characters are found.'''
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	322 s.decode('ascii')
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	323 return s.upper()
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	324
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	325 def asciiupper(s):
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	326 # delay importing avoids cyclic dependency around "parsers" in
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	327 # pure Python build (util => i18n => encoding => parsers => util)
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	328 from . import parsers
24578 ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	329 impl = getattr(parsers, 'asciiupper', _asciiupper)
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	330 global asciiupper
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	331 asciiupper = impl
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	332 return impl(s)
ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	333
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	334 def lower(s):
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	335 "best-effort encoding-aware case-folding of local string s"
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	336 try:
22779 d9585dda63c3 encoding.lower: use fast ASCII lower Siddharth Agarwal <sid0@fb.com> parents: 22778 diff changeset	337 return asciilower(s)
17235 3745ae495ce5 encoding: use s.decode to trigger UnicodeDecodeError Martin Geisler <mg@aragost.com> parents: 16493 diff changeset	338 except UnicodeDecodeError:
16387 c481761033bd encoding: add fast-path for ASCII lowercase Matt Mackall <mpm@selenic.com> parents: 16274 diff changeset	339 pass
c481761033bd encoding: add fast-path for ASCII lowercase Matt Mackall <mpm@selenic.com> parents: 16274 diff changeset	340 try:
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	341 if isinstance(s, localstr):
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	342 u = s._utf8.decode("utf-8")
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	343 else:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	344 u = s.decode(_sysstr(encoding), _sysstr(encodingmode))
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	345
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	346 lu = u.lower()
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	347 if u == lu:
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	348 return s # preserve localstring
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	349 return lu.encode(_sysstr(encoding))
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	350 except UnicodeError:
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	351 return s.lower() # we don't know how to fold this except in ASCII
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	352 except LookupError as k:
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	353 raise error.Abort(k, hint="please check your locale settings")
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	354
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	355 def upper(s):
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	356 "best-effort encoding-aware case-folding of local string s"
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	357 try:
24578 ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	358 return asciiupper(s)
17236 9fb8312dbdbd encoding: add fast-path for ASCII uppercase. Martin Geisler <mg@aragost.com> parents: 17235 diff changeset	359 except UnicodeDecodeError:
24597 b4258d5a1600 encoding.upper: factor out fallback code Siddharth Agarwal <sid0@fb.com> parents: 24593 diff changeset	360 return upperfallback(s)
b4258d5a1600 encoding.upper: factor out fallback code Siddharth Agarwal <sid0@fb.com> parents: 24593 diff changeset	361
b4258d5a1600 encoding.upper: factor out fallback code Siddharth Agarwal <sid0@fb.com> parents: 24593 diff changeset	362 def upperfallback(s):
17236 9fb8312dbdbd encoding: add fast-path for ASCII uppercase. Martin Geisler <mg@aragost.com> parents: 17235 diff changeset	363 try:
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	364 if isinstance(s, localstr):
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	365 u = s._utf8.decode("utf-8")
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	366 else:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	367 u = s.decode(_sysstr(encoding), _sysstr(encodingmode))
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	368
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	369 uu = u.upper()
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	370 if u == uu:
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	371 return s # preserve localstring
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	372 return uu.encode(_sysstr(encoding))
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	373 except UnicodeError:
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	374 return s.upper() # we don't know how to fold this except in ASCII
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	375 except LookupError as k:
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	376 raise error.Abort(k, hint="please check your locale settings")
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	377
24593 f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	378 class normcasespecs(object):
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	379 '''what a platform's normcase does to ASCII strings
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	380
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	381 This is specified per platform, and should be consistent with what normcase
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	382 on that platform actually does.
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	383
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	384 lower: normcase lowercases ASCII strings
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	385 upper: normcase uppercases ASCII strings
24608 1c533e23ce95 util.h: define an enum for normcase specs Siddharth Agarwal <sid0@fb.com> parents: 24597 diff changeset	386 other: the fallback function should always be called
1c533e23ce95 util.h: define an enum for normcase specs Siddharth Agarwal <sid0@fb.com> parents: 24597 diff changeset	387
1c533e23ce95 util.h: define an enum for normcase specs Siddharth Agarwal <sid0@fb.com> parents: 24597 diff changeset	388 This should be kept in sync with normcase_spec in util.h.'''
24593 f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	389 lower = -1
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	390 upper = 1
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	391 other = 0
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	392
28066 d1cc07123243 encoding: change jsonmap to a list indexed by code point Yuya Nishihara <yuya@tcha.org> parents: 27881 diff changeset	393 _jsonmap = []
28508 3c6e94d0811c encoding: use range() instead of xrange() Gregory Szorc <gregory.szorc@gmail.com> parents: 28507 diff changeset	394 _jsonmap.extend("\\u%04x" % x for x in range(32))
3c6e94d0811c encoding: use range() instead of xrange() Gregory Szorc <gregory.szorc@gmail.com> parents: 28507 diff changeset	395 _jsonmap.extend(chr(x) for x in range(32, 127))
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	396 _jsonmap.append('\\u007f')
28067 69a02b1e947c encoding: initialize jsonmap when module is loaded Yuya Nishihara <yuya@tcha.org> parents: 28066 diff changeset	397 _jsonmap[0x09] = '\\t'
69a02b1e947c encoding: initialize jsonmap when module is loaded Yuya Nishihara <yuya@tcha.org> parents: 28066 diff changeset	398 _jsonmap[0x0a] = '\\n'
69a02b1e947c encoding: initialize jsonmap when module is loaded Yuya Nishihara <yuya@tcha.org> parents: 28066 diff changeset	399 _jsonmap[0x22] = '\\"'
69a02b1e947c encoding: initialize jsonmap when module is loaded Yuya Nishihara <yuya@tcha.org> parents: 28066 diff changeset	400 _jsonmap[0x5c] = '\\\\'
69a02b1e947c encoding: initialize jsonmap when module is loaded Yuya Nishihara <yuya@tcha.org> parents: 28066 diff changeset	401 _jsonmap[0x08] = '\\b'
69a02b1e947c encoding: initialize jsonmap when module is loaded Yuya Nishihara <yuya@tcha.org> parents: 28066 diff changeset	402 _jsonmap[0x0c] = '\\f'
69a02b1e947c encoding: initialize jsonmap when module is loaded Yuya Nishihara <yuya@tcha.org> parents: 28066 diff changeset	403 _jsonmap[0x0d] = '\\r'
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	404 _paranoidjsonmap = _jsonmap[:]
28069 b2d24c2898f9 encoding: backport paranoid escaping from templatefilters.jsonescape() Yuya Nishihara <yuya@tcha.org> parents: 28068 diff changeset	405 _paranoidjsonmap[0x3c] = '\\u003c' # '<' (e.g. escape "</script>")
b2d24c2898f9 encoding: backport paranoid escaping from templatefilters.jsonescape() Yuya Nishihara <yuya@tcha.org> parents: 28068 diff changeset	406 _paranoidjsonmap[0x3e] = '\\u003e' # '>'
28508 3c6e94d0811c encoding: use range() instead of xrange() Gregory Szorc <gregory.szorc@gmail.com> parents: 28507 diff changeset	407 _jsonmap.extend(chr(x) for x in range(128, 256))
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	408
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	409 def jsonescape(s, paranoid=False):
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	410 '''returns a string suitable for JSON
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	411
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	412 JSON is problematic for us because it doesn't support non-Unicode
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	413 bytes. To deal with this, we take the following approach:
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	414
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	415 - localstr objects are converted back to UTF-8
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	416 - valid UTF-8/ASCII strings are passed as-is
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	417 - other strings are converted to UTF-8b surrogate encoding
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	418 - apply JSON-specified string escaping
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	419
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	420 (escapes are doubled in these tests)
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	421
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	422 >>> jsonescape('this is a test')
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	423 'this is a test'
27881 ffa599f3f503 encoding: escape U+007F (DEL) character in JSON Yuya Nishihara <yuya@tcha.org> parents: 27699 diff changeset	424 >>> jsonescape('escape characters: \\0 \\x0b \\x7f')
ffa599f3f503 encoding: escape U+007F (DEL) character in JSON Yuya Nishihara <yuya@tcha.org> parents: 27699 diff changeset	425 'escape characters: \\\\u0000 \\\\u000b \\\\u007f'
ffa599f3f503 encoding: escape U+007F (DEL) character in JSON Yuya Nishihara <yuya@tcha.org> parents: 27699 diff changeset	426 >>> jsonescape('escape characters: \\t \\n \\r \\" \\\\')
ffa599f3f503 encoding: escape U+007F (DEL) character in JSON Yuya Nishihara <yuya@tcha.org> parents: 27699 diff changeset	427 'escape characters: \\\\t \\\\n \\\\r \\\\" \\\\\\\\'
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	428 >>> jsonescape('a weird byte: \\xdd')
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	429 'a weird byte: \\xed\\xb3\\x9d'
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	430 >>> jsonescape('utf-8: caf\\xc3\\xa9')
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	431 'utf-8: caf\\xc3\\xa9'
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	432 >>> jsonescape('')
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	433 ''
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	434
28069 b2d24c2898f9 encoding: backport paranoid escaping from templatefilters.jsonescape() Yuya Nishihara <yuya@tcha.org> parents: 28068 diff changeset	435 If paranoid, non-ascii and common troublesome characters are also escaped.
b2d24c2898f9 encoding: backport paranoid escaping from templatefilters.jsonescape() Yuya Nishihara <yuya@tcha.org> parents: 28068 diff changeset	436 This is suitable for web output.
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	437
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	438 >>> jsonescape('escape boundary: \\x7e \\x7f \\xc2\\x80', paranoid=True)
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	439 'escape boundary: ~ \\\\u007f \\\\u0080'
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	440 >>> jsonescape('a weird byte: \\xdd', paranoid=True)
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	441 'a weird byte: \\\\udcdd'
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	442 >>> jsonescape('utf-8: caf\\xc3\\xa9', paranoid=True)
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	443 'utf-8: caf\\\\u00e9'
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	444 >>> jsonescape('non-BMP: \\xf0\\x9d\\x84\\x9e', paranoid=True)
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	445 'non-BMP: \\\\ud834\\\\udd1e'
28069 b2d24c2898f9 encoding: backport paranoid escaping from templatefilters.jsonescape() Yuya Nishihara <yuya@tcha.org> parents: 28068 diff changeset	446 >>> jsonescape('<foo@example.org>', paranoid=True)
b2d24c2898f9 encoding: backport paranoid escaping from templatefilters.jsonescape() Yuya Nishihara <yuya@tcha.org> parents: 28068 diff changeset	447 '\\\\u003cfoo@example.org\\\\u003e'
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	448 '''
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	449
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	450 if paranoid:
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	451 jm = _paranoidjsonmap
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	452 else:
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	453 jm = _jsonmap
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	454
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	455 u8chars = toutf8b(s)
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	456 try:
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	457 return ''.join(jm[x] for x in bytearray(u8chars)) # fast path
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	458 except IndexError:
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	459 pass
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	460 # non-BMP char is represented as UTF-16 surrogate pair
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	461 u16codes = array.array('H', u8chars.decode('utf-8').encode('utf-16'))
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	462 u16codes.pop(0) # drop BOM
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	463 return ''.join(jm[x] if x < 128 else '\\u%04x' % x for x in u16codes)
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	464
26875 cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	465 _utf8len = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 4]
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	466
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	467 def getutf8char(s, pos):
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	468 '''get the next full utf-8 character in the given string, starting at pos
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	469
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	470 Raises a UnicodeError if the given location does not start a valid
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	471 utf-8 character.
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	472 '''
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	473
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	474 # find how many bytes to attempt decoding from first nibble
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	475 l = _utf8len[ord(s[pos]) >> 4]
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	476 if not l: # ascii
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	477 return s[pos]
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	478
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	479 c = s[pos:pos + l]
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	480 # validate with attempted decode
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	481 c.decode("utf-8")
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	482 return c
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	483
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	484 def toutf8b(s):
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	485 '''convert a local, possibly-binary string into UTF-8b
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	486
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	487 This is intended as a generic method to preserve data when working
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	488 with schemes like JSON and XML that have no provision for
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	489 arbitrary byte strings. As Mercurial often doesn't know
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	490 what encoding data is in, we use so-called UTF-8b.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	491
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	492 If a string is already valid UTF-8 (or ASCII), it passes unmodified.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	493 Otherwise, unsupported bytes are mapped to UTF-16 surrogate range,
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	494 uDC00-uDCFF.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	495
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	496 Principles of operation:
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	497
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17236 diff changeset	498 - ASCII and UTF-8 data successfully round-trips and is understood
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	499 by Unicode-oriented clients
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	500 - filenames and file contents in arbitrary other encodings can have
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	501 be round-tripped or recovered by clueful clients
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	502 - local strings that have a cached known UTF-8 encoding (aka
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	503 localstr) get sent as UTF-8 so Unicode-oriented clients get the
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	504 Unicode data they want
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	505 - because we must preserve UTF-8 bytestring in places such as
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	506 filenames, metadata can't be roundtripped without help
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	507
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	508 (Note: "UTF-8b" often refers to decoding a mix of valid UTF-8 and
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	509 arbitrary bytes into an internal Unicode format that can be
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	510 re-encoded back into the original. Here we are exposing the
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	511 internal surrogate encoding as a UTF-8 string.)
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	512 '''
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	513
26879 a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	514 if "\xed" not in s:
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	515 if isinstance(s, localstr):
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	516 return s._utf8
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	517 try:
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	518 s.decode('utf-8')
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	519 return s
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	520 except UnicodeDecodeError:
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	521 pass
26878 d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	522
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	523 r = ""
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	524 pos = 0
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	525 l = len(s)
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	526 while pos < l:
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	527 try:
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	528 c = getutf8char(s, pos)
26879 a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	529 if "\xed\xb0\x80" <= c <= "\xed\xb3\xbf":
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	530 # have to re-escape existing U+DCxx characters
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	531 c = unichr(0xdc00 + ord(s[pos])).encode('utf-8')
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	532 pos += 1
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	533 else:
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	534 pos += len(c)
26878 d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	535 except UnicodeDecodeError:
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	536 c = unichr(0xdc00 + ord(s[pos])).encode('utf-8')
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	537 pos += 1
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	538 r += c
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	539 return r
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	540
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	541 def fromutf8b(s):
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	542 '''Given a UTF-8b string, return a local, possibly-binary string.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	543
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	544 return the original binary string. This
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	545 is a round-trip process for strings like filenames, but metadata
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	546 that's was passed through tolocal will remain in UTF-8.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	547
26963 de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	548 >>> roundtrip = lambda x: fromutf8b(toutf8b(x)) == x
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	549 >>> m = "\\xc3\\xa9\\x99abcd"
26963 de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	550 >>> toutf8b(m)
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	551 '\\xc3\\xa9\\xed\\xb2\\x99abcd'
26963 de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	552 >>> roundtrip(m)
de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	553 True
de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	554 >>> roundtrip("\\xc2\\xc2\\x80")
de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	555 True
de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	556 >>> roundtrip("\\xef\\xbf\\xbd")
de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	557 True
de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	558 >>> roundtrip("\\xef\\xef\\xbf\\xbd")
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	559 True
27699 c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	560 >>> roundtrip("\\xf1\\x80\\x80\\x80\\x80")
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	561 True
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	562 '''
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	563
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	564 # fast path - look for uDxxx prefixes in s
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	565 if "\xed" not in s:
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	566 return s
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	567
27699 c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	568 # We could do this with the unicode type but some Python builds
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	569 # use UTF-16 internally (issue5031) which causes non-BMP code
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	570 # points to be escaped. Instead, we use our handy getutf8char
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	571 # helper again to walk the string without "decoding" it.
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	572
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	573 r = ""
27699 c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	574 pos = 0
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	575 l = len(s)
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	576 while pos < l:
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	577 c = getutf8char(s, pos)
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	578 pos += len(c)
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	579 # unescape U+DCxx characters
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	580 if "\xed\xb0\x80" <= c <= "\xed\xb3\xbf":
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	581 c = chr(ord(c.decode("utf-8")) & 0xff)
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	582 r += c
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	583 return r

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/encoding.py @ 30033:02dbfaa6df0b