mercurial-scm/hg: mercurial/encoding.py annotate

annotate mercurial/encoding.py @ 15063:c20688b7c061 stable

setdiscovery: fix hang when #heads>200 (issue2971) When setting up the next sample, we always add all of the heads, regardless of the desired max sample size. But if the number of heads exceeds this size, then we don't add any more nodes from the still undecided set. (This is debatable per se, and I'll investigate it, but it's how we designed it at the moment.) The bug was that we always added the overall heads, not the heads of the remaining undecided set. Thus, if #heads>200 (desired sample size), we did not make progress any longer.

author	Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
date	Thu, 25 Aug 2011 21:25:14 +0200
parents	e38846a79a23
children	61807854004e

rev	line source
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	1 # encoding.py - character transcoding support for Mercurial
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	3 # Copyright 2005-2009 Matt Mackall <mpm@selenic.com> and others
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	4 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 9574 diff changeset	6 # GNU General Public License version 2 or any later version.
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	7
8312 b87a50b7125c separate import lines from mercurial and general python modules Simon Heimberg <simohe@besonet.ch> parents: 8226 diff changeset	8 import error
12062 c327bfa5e831 cleanup: remove unused imports Brodie Rao <brodie@bitheap.org> parents: 11892 diff changeset	9 import unicodedata, locale, os
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	10
11892 2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	11 def _getpreferredencoding():
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	12 '''
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	13 On darwin, getpreferredencoding ignores the locale environment and
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	14 always returns mac-roman. http://bugs.python.org/issue6202 fixes this
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	15 for Python 2.7 and up. This is the same corrected code for earlier
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	16 Python versions.
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	17
12770 614f0d8724ab check-code: find trailing whitespace Martin Geisler <mg@lazybytes.net> parents: 12062 diff changeset	18 However, we can't use a version check for this method, as some distributions
11892 2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	19 patch Python to fix this. Instead, we use it as a 'fixer' for the mac-roman
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	20 encoding, as it is unlikely that this encoding is the actually expected.
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	21 '''
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	22 try:
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	23 locale.CODESET
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	24 except AttributeError:
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	25 # Fall back to parsing environment variables :-(
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	26 return locale.getdefaultlocale()[1]
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	27
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	28 oldloc = locale.setlocale(locale.LC_CTYPE)
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	29 locale.setlocale(locale.LC_CTYPE, "")
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	30 result = locale.nl_langinfo(locale.CODESET)
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	31 locale.setlocale(locale.LC_CTYPE, oldloc)
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	32
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	33 return result
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	34
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	35 _encodingfixers = {
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	36 '646': lambda: 'ascii',
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	37 'ANSI_X3.4-1968': lambda: 'ascii',
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	38 'mac-roman': _getpreferredencoding
2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	39 }
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	40
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	41 try:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	42 encoding = os.environ.get("HGENCODING")
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	43 if not encoding:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	44 encoding = locale.getpreferredencoding() or 'ascii'
11892 2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	45 encoding = _encodingfixers.get(encoding, lambda: encoding)()
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	46 except locale.Error:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	47 encoding = 'ascii'
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	48 encodingmode = os.environ.get("HGENCODINGMODE", "strict")
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	49 fallbackencoding = 'ISO-8859-1'
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	50
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	51 class localstr(str):
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	52 '''This class allows strings that are unmodified to be
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	53 round-tripped to the local encoding and back'''
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	54 def __new__(cls, u, l):
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	55 s = str.__new__(cls, l)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	56 s._utf8 = u
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	57 return s
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	58 def __hash__(self):
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	59 return hash(self._utf8) # avoid collisions in local string space
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	60
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	61 def tolocal(s):
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	62 """
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	63 Convert a string from internal UTF-8 to local encoding
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	64
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	65 All internal strings should be UTF-8 but some repos before the
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	66 implementation of locale support may contain latin1 or possibly
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	67 other character sets. We attempt to decode everything strictly
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	68 using UTF-8, then Latin-1, and failing that, we use UTF-8 and
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	69 replace unknown characters.
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	70
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	71 The localstr class is used to cache the known UTF-8 encoding of
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	72 strings next to their local representation to allow lossless
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	73 round-trip conversion back to UTF-8.
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	74
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	75 >>> u = 'foo: \\xc3\\xa4' # utf-8
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	76 >>> l = tolocal(u)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	77 >>> l
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	78 'foo: ?'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	79 >>> fromlocal(l)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	80 'foo: \\xc3\\xa4'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	81 >>> u2 = 'foo: \\xc3\\xa1'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	82 >>> d = { l: 1, tolocal(u2): 2 }
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	83 >>> d # no collision
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	84 {'foo: ?': 1, 'foo: ?': 2}
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	85 >>> 'foo: ?' in d
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	86 False
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	87 >>> l1 = 'foo: \\xe4' # historical latin1 fallback
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	88 >>> l = tolocal(l1)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	89 >>> l
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	90 'foo: ?'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	91 >>> fromlocal(l) # magically in utf-8
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	92 'foo: \\xc3\\xa4'
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	93 """
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	94
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	95 for e in ('UTF-8', fallbackencoding):
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	96 try:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	97 u = s.decode(e) # attempt strict decoding
13940 b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	98 r = u.encode(encoding, "replace")
b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	99 if u == r.decode(encoding):
b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	100 # r is a safe, non-lossy encoding of s
b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	101 return r
b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	102 elif e == 'UTF-8':
b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	103 return localstr(s, r)
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	104 else:
13940 b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	105 return localstr(u.encode('UTF-8'), r)
b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	106
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	107 except LookupError, k:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	108 raise error.Abort("%s, please check your locale settings" % k)
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	109 except UnicodeDecodeError:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	110 pass
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	111 u = s.decode("utf-8", "replace") # last ditch
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	112 return u.encode(encoding, "replace") # can't round-trip
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	113
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	114 def fromlocal(s):
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	115 """
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	116 Convert a string from the local character encoding to UTF-8
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	117
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	118 We attempt to decode strings using the encoding mode set by
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	119 HGENCODINGMODE, which defaults to 'strict'. In this mode, unknown
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	120 characters will cause an error message. Other modes include
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	121 'replace', which replaces unknown characters with a special
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	122 Unicode character, and 'ignore', which drops the character.
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	123 """
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	124
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	125 # can we do a lossless round-trip?
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	126 if isinstance(s, localstr):
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	127 return s._utf8
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	128
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	129 try:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	130 return s.decode(encoding, encodingmode).encode("utf-8")
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	131 except UnicodeDecodeError, inst:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 10263 diff changeset	132 sub = s[max(0, inst.start - 10):inst.start + 10]
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	133 raise error.Abort("decoding near '%s': %s!" % (sub, inst))
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	134 except LookupError, k:
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	135 raise error.Abort("%s, please check your locale settings" % k)
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	136
12866 eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	137 # How to treat ambiguous-width characters. Set to 'wide' to treat as wide.
eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	138 ambiguous = os.environ.get("HGENCODINGAMBIGUOUS", "narrow")
eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	139
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	140 def colwidth(s):
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	141 "Find the column width of a UTF-8 string for display"
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	142 d = s.decode(encoding, 'replace')
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	143 if hasattr(unicodedata, 'east_asian_width'):
12866 eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	144 wide = "WF"
eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	145 if ambiguous == "wide":
eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	146 wide = "WFA"
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	147 w = unicodedata.east_asian_width
12866 eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	148 return sum([w(c) in wide and 2 or 1 for c in d])
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	149 return len(d)
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	150
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	151 def lower(s):
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	152 "best-effort encoding-aware case-folding of local string s"
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	153 try:
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	154 if isinstance(s, localstr):
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	155 u = s._utf8.decode("utf-8")
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	156 else:
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	157 u = s.decode(encoding, encodingmode)
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	158
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	159 lu = u.lower()
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	160 if u == lu:
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	161 return s # preserve localstring
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	162 return lu.encode(encoding)
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	163 except UnicodeError:
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	164 return s.lower() # we don't know how to fold this except in ASCII

Mercurial > public > mercurial-scm > hg

annotate mercurial/encoding.py @ 15063:c20688b7c061 stable