mercurial-scm/hg-stable: mercurial/encoding.py annotate

annotate mercurial/encoding.py @ 43725:da925257a39e

typing: add pseudo localstr.__init__() to help pytype Apparently, pytype failed to parse localstr.__new__()? This fixes the following errors: line 126, in __hash__: No attribute '_utf8' on localstr [attribute-error] line 188, in tolocal: Function localstr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, string: str, ...) Actually passed: (self, string: bytes, ...)

author	Yuya Nishihara <yuya@tcha.org>
date	Sat, 16 Nov 2019 16:25:28 +0900
parents	009c115eba95
children	7b14d649af1b

rev	line source
8226 8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	1 # encoding.py - character transcoding support for Mercurial
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	2 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	3 # Copyright 2005-2009 Matt Mackall <mpm@selenic.com> and others
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	4 #
8b2cd04a6e97 put license and copyright info into comment blocks Martin Geisler <mg@lazybytes.net> parents: 8225 diff changeset	5 # This software may be used and distributed according to the terms of the
10263 25e572394f5c Update license to GPLv2+ Matt Mackall <mpm@selenic.com> parents: 9574 diff changeset	6 # GNU General Public License version 2 or any later version.
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	7
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	8 from __future__ import absolute_import, print_function
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	9
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	10 import locale
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	11 import os
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	12 import unicodedata
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	13
43089 c59eb1560c44 py3: manually import getattr where it is needed Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	14 from .pycompat import getattr
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	15 from . import (
b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	16 error,
32411 df448de7cf3b parsers: switch to policy importer Yuya Nishihara <yuya@tcha.org> parents: 32339 diff changeset	17 policy,
30031 0f6d6fdd3c2a pycompat: provide 'ispy3' constant Yuya Nishihara <yuya@tcha.org> parents: 28508 diff changeset	18 pycompat,
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	19 )
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	20
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	21 from .pure import charencode as charencodepure
33942 b9101467d88b encoding: extract stub for fast JSON escape Yuya Nishihara <yuya@tcha.org> parents: 33873 diff changeset	22
43725 da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	23 _TYPE_CHECKING = False
da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	24
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	25 if not globals(): # hide this from non-pytype users
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	26 from typing import (
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	27 Any,
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	28 Callable,
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	29 List,
43725 da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	30 TYPE_CHECKING as _TYPE_CHECKING,
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	31 Text,
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	32 Type,
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	33 TypeVar,
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	34 Union,
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	35 )
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	36
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	37 # keep pyflakes happy
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	38 for t in (Any, Callable, List, Text, Type, Union):
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	39 assert t
2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	40
43720 3364a15f61f0 typing: fix forward reference in _Tlocalstr type bound Yuya Nishihara <yuya@tcha.org> parents: 43719 diff changeset	41 _Tlocalstr = TypeVar('_Tlocalstr', bound='localstr')
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	42
43554 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43551 diff changeset	43 charencode = policy.importmod('charencode')
33782 f5fc54e7e467 encoding: drop circular import by proxying through '<policy>.charencode' Yuya Nishihara <yuya@tcha.org> parents: 33038 diff changeset	44
33944 f4433f2713d0 encoding: add function to test if a str consists of ASCII characters Yuya Nishihara <yuya@tcha.org> parents: 33943 diff changeset	45 isasciistr = charencode.isasciistr
33782 f5fc54e7e467 encoding: drop circular import by proxying through '<policy>.charencode' Yuya Nishihara <yuya@tcha.org> parents: 33038 diff changeset	46 asciilower = charencode.asciilower
f5fc54e7e467 encoding: drop circular import by proxying through '<policy>.charencode' Yuya Nishihara <yuya@tcha.org> parents: 33038 diff changeset	47 asciiupper = charencode.asciiupper
33943 2c37f9dabc32 encoding: add fast path of jsonescape() (issue5533) Yuya Nishihara <yuya@tcha.org> parents: 33942 diff changeset	48 _jsonescapeu8fast = charencode.jsonescapeu8fast
33782 f5fc54e7e467 encoding: drop circular import by proxying through '<policy>.charencode' Yuya Nishihara <yuya@tcha.org> parents: 33038 diff changeset	49
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	50 _sysstr = pycompat.sysstr
02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	51
30031 0f6d6fdd3c2a pycompat: provide 'ispy3' constant Yuya Nishihara <yuya@tcha.org> parents: 28508 diff changeset	52 if pycompat.ispy3:
28507 9bcbd9412225 encoding: make HFS+ ignore code Python 3 compatible Gregory Szorc <gregory.szorc@gmail.com> parents: 28069 diff changeset	53 unichr = chr
9bcbd9412225 encoding: make HFS+ ignore code Python 3 compatible Gregory Szorc <gregory.szorc@gmail.com> parents: 28069 diff changeset	54
23596 885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	55 # These unicode characters are ignored by HFS+ (Apple Technote 1150,
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	56 # "Unicode Subtleties"), so we need to ignore them in some places for
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	57 # sanity.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	58 _ignore = [
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	59 unichr(int(x, 16)).encode("utf-8")
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	60 for x in b"200c 200d 200e 200f 202a 202b 202c 202d 202e "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	61 b"206a 206b 206c 206d 206e 206f feff".split()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	62 ]
23596 885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	63 # verify the next function will work
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	64 assert all(i.startswith((b"\xe2", b"\xef")) for i in _ignore)
23596 885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	65
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	66
23596 885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	67 def hfsignoreclean(s):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	68 # type: (bytes) -> bytes
23596 885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	69 """Remove codepoints ignored by HFS+ from s.
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	70
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	71 >>> hfsignoreclean(u'.h\u200cg'.encode('utf-8'))
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	72 '.hg'
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	73 >>> hfsignoreclean(u'.h\ufeffg'.encode('utf-8'))
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	74 '.hg'
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	75 """
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	76 if b"\xe2" in s or b"\xef" in s:
23596 885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	77 for c in _ignore:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	78 s = s.replace(c, b'')
23596 885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	79 return s
885bd7c5c7e3 encoding: add hfsignoreclean to clean out HFS-ignored characters Augie Fackler <raf@durin42.com> parents: 22973 diff changeset	80
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	81
30034 e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	82 # encoding.environ is provided read-only, which may not be used to modify
e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	83 # the process environment
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	84 _nativeenviron = not pycompat.ispy3 or os.supports_bytes_environ
30034 e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	85 if not pycompat.ispy3:
32231 cf424dae5dc7 check-code: ignore re-exports of os.environ in encoding.py Yuya Nishihara <yuya@tcha.org> parents: 32205 diff changeset	86 environ = os.environ # re-exports
30034 e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	87 elif _nativeenviron:
32231 cf424dae5dc7 check-code: ignore re-exports of os.environ in encoding.py Yuya Nishihara <yuya@tcha.org> parents: 32205 diff changeset	88 environ = os.environb # re-exports
30034 e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	89 else:
e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	90 # preferred encoding isn't known yet; use utf-8 to avoid unicode error
e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	91 # and recreate it once encoding is settled
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	92 environ = dict(
43554 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43551 diff changeset	93 (k.encode('utf-8'), v.encode('utf-8'))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	94 for k, v in os.environ.items() # re-exports
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	95 )
30034 e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	96
39844 9e8fcd2e78c1 encoding: remove unnecessary lambdas from _encodingfixers Martin von Zweigbergk <martinvonz@google.com> parents: 39824 diff changeset	97 _encodingrewrites = {
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	98 b'646': b'ascii',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	99 b'ANSI_X3.4-1968': b'ascii',
11892 2be70ca17311 encoding: improve handling of buggy getpreferredencoding() on Mac OS X Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 11297 diff changeset	100 }
37883 443029011990 encoding: alias cp65001 to utf-8 on Windows Yuya Nishihara <yuya@tcha.org> parents: 36805 diff changeset	101 # cp65001 is a Windows variant of utf-8, which isn't supported on Python 2.
443029011990 encoding: alias cp65001 to utf-8 on Windows Yuya Nishihara <yuya@tcha.org> parents: 36805 diff changeset	102 # No idea if it should be rewritten to the canonical name 'utf-8' on Python 3.
443029011990 encoding: alias cp65001 to utf-8 on Windows Yuya Nishihara <yuya@tcha.org> parents: 36805 diff changeset	103 # https://bugs.python.org/issue13216
443029011990 encoding: alias cp65001 to utf-8 on Windows Yuya Nishihara <yuya@tcha.org> parents: 36805 diff changeset	104 if pycompat.iswindows and not pycompat.ispy3:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	105 _encodingrewrites[b'cp65001'] = b'utf-8'
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	106
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	107 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	108 encoding = environ.get(b"HGENCODING")
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	109 if not encoding:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	110 encoding = locale.getpreferredencoding().encode('ascii') or b'ascii'
39844 9e8fcd2e78c1 encoding: remove unnecessary lambdas from _encodingfixers Martin von Zweigbergk <martinvonz@google.com> parents: 39824 diff changeset	111 encoding = _encodingrewrites.get(encoding, encoding)
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	112 except locale.Error:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	113 encoding = b'ascii'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	114 encodingmode = environ.get(b"HGENCODINGMODE", b"strict")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	115 fallbackencoding = b'ISO-8859-1'
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	116
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	117
33832 dabe1f11ae3a py3: change encoding.localstr to a subclass of bytes, not str Yuya Nishihara <yuya@tcha.org> parents: 33782 diff changeset	118 class localstr(bytes):
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	119 '''This class allows strings that are unmodified to be
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	120 round-tripped to the local encoding and back'''
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	121
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	122 def __new__(cls, u, l):
33832 dabe1f11ae3a py3: change encoding.localstr to a subclass of bytes, not str Yuya Nishihara <yuya@tcha.org> parents: 33782 diff changeset	123 s = bytes.__new__(cls, l)
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	124 s._utf8 = u
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	125 return s
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	126
43725 da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	127 if _TYPE_CHECKING:
da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	128 # pseudo implementation to help pytype see localstr() constructor
da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	129 def __init__(self, u, l):
da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	130 # type: (bytes, bytes) -> None
da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	131 super(localstr, self).__init__(l)
da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	132 self._utf8 = u
da925257a39e typing: add pseudo localstr.__init__() to help pytype Yuya Nishihara <yuya@tcha.org> parents: 43724 diff changeset	133
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	134 def __hash__(self):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	135 return hash(self._utf8) # avoid collisions in local string space
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	136
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	137
37991 3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	138 class safelocalstr(bytes):
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	139 """Tagged string denoting it was previously an internal UTF-8 string,
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	140 and can be converted back to UTF-8 losslessly
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	141
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	142 >>> assert safelocalstr(b'\\xc3') == b'\\xc3'
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	143 >>> assert b'\\xc3' == safelocalstr(b'\\xc3')
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	144 >>> assert b'\\xc3' in {safelocalstr(b'\\xc3'): 0}
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	145 >>> assert safelocalstr(b'\\xc3') in {b'\\xc3': 0}
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	146 """
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	147
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	148
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	149 def tolocal(s):
43721 b65fcccd9100 typing: fix argument type of encoding.tolocal() and .fromutf8b() Yuya Nishihara <yuya@tcha.org> parents: 43720 diff changeset	150 # type: (bytes) -> bytes
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	151 """
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	152 Convert a string from internal UTF-8 to local encoding
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	153
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	154 All internal strings should be UTF-8 but some repos before the
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	155 implementation of locale support may contain latin1 or possibly
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	156 other character sets. We attempt to decode everything strictly
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	157 using UTF-8, then Latin-1, and failing that, we use UTF-8 and
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	158 replace unknown characters.
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	159
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	160 The localstr class is used to cache the known UTF-8 encoding of
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	161 strings next to their local representation to allow lossless
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	162 round-trip conversion back to UTF-8.
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	163
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	164 >>> u = b'foo: \\xc3\\xa4' # utf-8
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	165 >>> l = tolocal(u)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	166 >>> l
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	167 'foo: ?'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	168 >>> fromlocal(l)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	169 'foo: \\xc3\\xa4'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	170 >>> u2 = b'foo: \\xc3\\xa1'
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	171 >>> d = { l: 1, tolocal(u2): 2 }
18378 404feac78b8a tests: stabilize doctest output Mads Kiilerich <mads@kiilerich.com> parents: 17424 diff changeset	172 >>> len(d) # no collision
404feac78b8a tests: stabilize doctest output Mads Kiilerich <mads@kiilerich.com> parents: 17424 diff changeset	173 2
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	174 >>> b'foo: ?' in d
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	175 False
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	176 >>> l1 = b'foo: \\xe4' # historical latin1 fallback
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	177 >>> l = tolocal(l1)
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	178 >>> l
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	179 'foo: ?'
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	180 >>> fromlocal(l) # magically in utf-8
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	181 'foo: \\xc3\\xa4'
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	182 """
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	183
33945 853574db5b12 encoding: add fast path of from/tolocal() for ASCII strings Yuya Nishihara <yuya@tcha.org> parents: 33944 diff changeset	184 if isasciistr(s):
853574db5b12 encoding: add fast path of from/tolocal() for ASCII strings Yuya Nishihara <yuya@tcha.org> parents: 33944 diff changeset	185 return s
853574db5b12 encoding: add fast path of from/tolocal() for ASCII strings Yuya Nishihara <yuya@tcha.org> parents: 33944 diff changeset	186
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	187 try:
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	188 try:
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	189 # make sure string is actually stored in UTF-8
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	190 u = s.decode('UTF-8')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	191 if encoding == b'UTF-8':
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	192 # fast path
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	193 return s
43551 313e3a279828 cleanup: remove pointless r-prefixes on double-quoted strings Augie Fackler <augie@google.com> parents: 43544 diff changeset	194 r = u.encode(_sysstr(encoding), "replace")
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	195 if u == r.decode(_sysstr(encoding)):
13940 b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	196 # r is a safe, non-lossy encoding of s
37991 3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	197 return safelocalstr(r)
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	198 return localstr(s, r)
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	199 except UnicodeDecodeError:
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	200 # we should only get here if we're looking at an ancient changeset
5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	201 try:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	202 u = s.decode(_sysstr(fallbackencoding))
43551 313e3a279828 cleanup: remove pointless r-prefixes on double-quoted strings Augie Fackler <augie@google.com> parents: 43544 diff changeset	203 r = u.encode(_sysstr(encoding), "replace")
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	204 if u == r.decode(_sysstr(encoding)):
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	205 # r is a safe, non-lossy encoding of s
37991 3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	206 return safelocalstr(r)
13940 b7b26e54e37a encoding: avoid localstr when a string can be encoded losslessly (issue2763) Matt Mackall <mpm@selenic.com> parents: 13051 diff changeset	207 return localstr(u.encode('UTF-8'), r)
16274 5d75eb8568d1 encoding: tune fast-path of tolocal a bit Matt Mackall <mpm@selenic.com> parents: 16133 diff changeset	208 except UnicodeDecodeError:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	209 u = s.decode("utf-8", "replace") # last ditch
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	210 # can't round-trip
43551 313e3a279828 cleanup: remove pointless r-prefixes on double-quoted strings Augie Fackler <augie@google.com> parents: 43544 diff changeset	211 return u.encode(_sysstr(encoding), "replace")
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	212 except LookupError as k:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	213 raise error.Abort(k, hint=b"please check your locale settings")
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	214
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	215
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	216 def fromlocal(s):
43681 7edc07fb890c encoding: fix bad type annotation Augie Fackler <augie@google.com> parents: 43554 diff changeset	217 # type: (bytes) -> bytes
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	218 """
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	219 Convert a string from the local character encoding to UTF-8
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	220
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	221 We attempt to decode strings using the encoding mode set by
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	222 HGENCODINGMODE, which defaults to 'strict'. In this mode, unknown
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	223 characters will cause an error message. Other modes include
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	224 'replace', which replaces unknown characters with a special
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	225 Unicode character, and 'ignore', which drops the character.
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	226 """
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	227
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	228 # can we do a lossless round-trip?
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	229 if isinstance(s, localstr):
7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	230 return s._utf8
33945 853574db5b12 encoding: add fast path of from/tolocal() for ASCII strings Yuya Nishihara <yuya@tcha.org> parents: 33944 diff changeset	231 if isasciistr(s):
853574db5b12 encoding: add fast path of from/tolocal() for ASCII strings Yuya Nishihara <yuya@tcha.org> parents: 33944 diff changeset	232 return s
13046 7cc4263e07a9 encoding: add localstr class to track UTF-8 version of transcoded strings Matt Mackall <mpm@selenic.com> parents: 12866 diff changeset	233
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	234 try:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	235 u = s.decode(_sysstr(encoding), _sysstr(encodingmode))
02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	236 return u.encode("utf-8")
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	237 except UnicodeDecodeError as inst:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	238 sub = s[max(0, inst.start - 10) : inst.start + 10]
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	239 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	240 b"decoding near '%s': %s!" % (sub, pycompat.bytestr(inst))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	241 )
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	242 except LookupError as k:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	243 raise error.Abort(k, hint=b"please check your locale settings")
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	244
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	245
31456 067add650129 encoding: factor out unicode variants of from/tolocal() Yuya Nishihara <yuya@tcha.org> parents: 30627 diff changeset	246 def unitolocal(u):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	247 # type: (Text) -> bytes
31456 067add650129 encoding: factor out unicode variants of from/tolocal() Yuya Nishihara <yuya@tcha.org> parents: 30627 diff changeset	248 """Convert a unicode string to a byte string of local encoding"""
067add650129 encoding: factor out unicode variants of from/tolocal() Yuya Nishihara <yuya@tcha.org> parents: 30627 diff changeset	249 return tolocal(u.encode('utf-8'))
067add650129 encoding: factor out unicode variants of from/tolocal() Yuya Nishihara <yuya@tcha.org> parents: 30627 diff changeset	250
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	251
31456 067add650129 encoding: factor out unicode variants of from/tolocal() Yuya Nishihara <yuya@tcha.org> parents: 30627 diff changeset	252 def unifromlocal(s):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	253 # type: (bytes) -> Text
31456 067add650129 encoding: factor out unicode variants of from/tolocal() Yuya Nishihara <yuya@tcha.org> parents: 30627 diff changeset	254 """Convert a byte string of local encoding to a unicode string"""
067add650129 encoding: factor out unicode variants of from/tolocal() Yuya Nishihara <yuya@tcha.org> parents: 30627 diff changeset	255 return fromlocal(s).decode('utf-8')
067add650129 encoding: factor out unicode variants of from/tolocal() Yuya Nishihara <yuya@tcha.org> parents: 30627 diff changeset	256
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	257
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32570 diff changeset	258 def unimethod(bytesfunc):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	259 # type: (Callable[[Any], bytes]) -> Callable[[Any], Text]
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32570 diff changeset	260 """Create a proxy method that forwards __unicode__() and __str__() of
ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32570 diff changeset	261 Python 3 to __bytes__()"""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	262
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32570 diff changeset	263 def unifunc(obj):
ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32570 diff changeset	264 return unifromlocal(bytesfunc(obj))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	265
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32570 diff changeset	266 return unifunc
ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32570 diff changeset	267
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	268
31457 6419cd243017 encoding: add converter between native str and byte string Yuya Nishihara <yuya@tcha.org> parents: 31456 diff changeset	269 # converter functions between native str and byte string. use these if the
6419cd243017 encoding: add converter between native str and byte string Yuya Nishihara <yuya@tcha.org> parents: 31456 diff changeset	270 # character encoding is not aware (e.g. exception message) or is known to
6419cd243017 encoding: add converter between native str and byte string Yuya Nishihara <yuya@tcha.org> parents: 31456 diff changeset	271 # be locale dependent (e.g. date formatting.)
6419cd243017 encoding: add converter between native str and byte string Yuya Nishihara <yuya@tcha.org> parents: 31456 diff changeset	272 if pycompat.ispy3:
6419cd243017 encoding: add converter between native str and byte string Yuya Nishihara <yuya@tcha.org> parents: 31456 diff changeset	273 strtolocal = unitolocal
6419cd243017 encoding: add converter between native str and byte string Yuya Nishihara <yuya@tcha.org> parents: 31456 diff changeset	274 strfromlocal = unifromlocal
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32570 diff changeset	275 strmethod = unimethod
31457 6419cd243017 encoding: add converter between native str and byte string Yuya Nishihara <yuya@tcha.org> parents: 31456 diff changeset	276 else:
43517 5f2a8dabb0d8 encoding: define local identify functions with explicit type comments Augie Fackler <augie@google.com> parents: 43089 diff changeset	277
5f2a8dabb0d8 encoding: define local identify functions with explicit type comments Augie Fackler <augie@google.com> parents: 43089 diff changeset	278 def strtolocal(s):
5f2a8dabb0d8 encoding: define local identify functions with explicit type comments Augie Fackler <augie@google.com> parents: 43089 diff changeset	279 # type: (str) -> bytes
43723 7f51bc36194d typing: suppress error of py2 encoding.strtolocal() and .strfromlocal() Yuya Nishihara <yuya@tcha.org> parents: 43722 diff changeset	280 return s # pytype: disable=bad-return-type
43517 5f2a8dabb0d8 encoding: define local identify functions with explicit type comments Augie Fackler <augie@google.com> parents: 43089 diff changeset	281
5f2a8dabb0d8 encoding: define local identify functions with explicit type comments Augie Fackler <augie@google.com> parents: 43089 diff changeset	282 def strfromlocal(s):
5f2a8dabb0d8 encoding: define local identify functions with explicit type comments Augie Fackler <augie@google.com> parents: 43089 diff changeset	283 # type: (bytes) -> str
43723 7f51bc36194d typing: suppress error of py2 encoding.strtolocal() and .strfromlocal() Yuya Nishihara <yuya@tcha.org> parents: 43722 diff changeset	284 return s # pytype: disable=bad-return-type
43517 5f2a8dabb0d8 encoding: define local identify functions with explicit type comments Augie Fackler <augie@google.com> parents: 43089 diff changeset	285
33038 ce96efec8112 py3: add utility to forward __str__() to __bytes__() Yuya Nishihara <yuya@tcha.org> parents: 32570 diff changeset	286 strmethod = pycompat.identity
31457 6419cd243017 encoding: add converter between native str and byte string Yuya Nishihara <yuya@tcha.org> parents: 31456 diff changeset	287
30034 e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	288 if not _nativeenviron:
e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	289 # now encoding and helper functions are available, recreate the environ
e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	290 # dict to be exported to other modules
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	291 environ = dict(
43554 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43551 diff changeset	292 (tolocal(k.encode('utf-8')), tolocal(v.encode('utf-8')))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	293 for k, v in os.environ.items() # re-exports
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	294 )
30034 e4a6b439acc5 py3: provide encoding.environ which is a dict of bytes Yuya Nishihara <yuya@tcha.org> parents: 30033 diff changeset	295
39823 24e493ec2229 py3: rename pycompat.getcwd() to encoding.getcwd() (API) Matt Harbison <matt_harbison@yahoo.com> parents: 38823 diff changeset	296 if pycompat.ispy3:
24e493ec2229 py3: rename pycompat.getcwd() to encoding.getcwd() (API) Matt Harbison <matt_harbison@yahoo.com> parents: 38823 diff changeset	297 # os.getcwd() on Python 3 returns string, but it has os.getcwdb() which
24e493ec2229 py3: rename pycompat.getcwd() to encoding.getcwd() (API) Matt Harbison <matt_harbison@yahoo.com> parents: 38823 diff changeset	298 # returns bytes.
39824 fb628c048d64 py3: don't use os.getcwdb() on Windows to avoid DeprecationWarnings Matt Harbison <matt_harbison@yahoo.com> parents: 39823 diff changeset	299 if pycompat.iswindows:
fb628c048d64 py3: don't use os.getcwdb() on Windows to avoid DeprecationWarnings Matt Harbison <matt_harbison@yahoo.com> parents: 39823 diff changeset	300 # Python 3 on Windows issues a DeprecationWarning about using the bytes
fb628c048d64 py3: don't use os.getcwdb() on Windows to avoid DeprecationWarnings Matt Harbison <matt_harbison@yahoo.com> parents: 39823 diff changeset	301 # API when os.getcwdb() is called.
fb628c048d64 py3: don't use os.getcwdb() on Windows to avoid DeprecationWarnings Matt Harbison <matt_harbison@yahoo.com> parents: 39823 diff changeset	302 getcwd = lambda: strtolocal(os.getcwd()) # re-exports
fb628c048d64 py3: don't use os.getcwdb() on Windows to avoid DeprecationWarnings Matt Harbison <matt_harbison@yahoo.com> parents: 39823 diff changeset	303 else:
fb628c048d64 py3: don't use os.getcwdb() on Windows to avoid DeprecationWarnings Matt Harbison <matt_harbison@yahoo.com> parents: 39823 diff changeset	304 getcwd = os.getcwdb # re-exports
39823 24e493ec2229 py3: rename pycompat.getcwd() to encoding.getcwd() (API) Matt Harbison <matt_harbison@yahoo.com> parents: 38823 diff changeset	305 else:
24e493ec2229 py3: rename pycompat.getcwd() to encoding.getcwd() (API) Matt Harbison <matt_harbison@yahoo.com> parents: 38823 diff changeset	306 getcwd = os.getcwd # re-exports
24e493ec2229 py3: rename pycompat.getcwd() to encoding.getcwd() (API) Matt Harbison <matt_harbison@yahoo.com> parents: 38823 diff changeset	307
12866 eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	308 # How to treat ambiguous-width characters. Set to 'wide' to treat as wide.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	309 _wide = _sysstr(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	310 environ.get(b"HGENCODINGAMBIGUOUS", b"narrow") == b"wide"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	311 and b"WFA"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	312 or b"WF"
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	313 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	314
12866 eddc20306ab6 encoding: default ambiguous character to narrow Matt Mackall <mpm@selenic.com> parents: 12770 diff changeset	315
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	316 def colwidth(s):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	317 # type: (bytes) -> int
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	318 b"Find the column width of a string for display in the local encoding"
43554 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43551 diff changeset	319 return ucolwidth(s.decode(_sysstr(encoding), 'replace'))
15066 24efa83d81cb i18n: calculate terminal columns by width information of each characters FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 14951 diff changeset	320
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	321
15066 24efa83d81cb i18n: calculate terminal columns by width information of each characters FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 14951 diff changeset	322 def ucolwidth(d):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	323 # type: (Text) -> int
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	324 b"Find the column width of a Unicode string for display"
14951 61807854004e encoding: use getattr isntead of hasattr Augie Fackler <durin42@gmail.com> parents: 14069 diff changeset	325 eaw = getattr(unicodedata, 'east_asian_width', None)
61807854004e encoding: use getattr isntead of hasattr Augie Fackler <durin42@gmail.com> parents: 14069 diff changeset	326 if eaw is not None:
32570 044f3d7eb9ae encoding: make sure "wide" variable never be referenced from other modules Yuya Nishihara <yuya@tcha.org> parents: 32562 diff changeset	327 return sum([eaw(c) in _wide and 2 or 1 for c in d])
7948 de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	328 return len(d)
de377b1a9a84 move encoding bits from util to encoding Matt Mackall <mpm@selenic.com> parents: diff changeset	329
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	330
15143 16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	331 def getcols(s, start, c):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	332 # type: (bytes, int, int) -> bytes
15143 16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	333 '''Use colwidth to find a c-column substring of s starting at byte
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	334 index start'''
38823 e7aa113b14f7 global: use pycompat.xrange() Gregory Szorc <gregory.szorc@gmail.com> parents: 38739 diff changeset	335 for x in pycompat.xrange(start + c, len(s)):
15143 16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	336 t = s[start:x]
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	337 if colwidth(t) == c:
16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	338 return t
43719 7cf332318f62 encoding: make getcols() raise exception explicitly Yuya Nishihara <yuya@tcha.org> parents: 43681 diff changeset	339 raise ValueError('substring not found')
15143 16c129b0f465 encoding: add getcols to extract substrings based on column width Matt Mackall <mpm@selenic.com> parents: 15142 diff changeset	340
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	341
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	342 def trim(s, width, ellipsis=b'', leftside=False):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	343 # type: (bytes, int, bytes, bool) -> bytes
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	344 """Trim string 's' to at most 'width' columns (including 'ellipsis').
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	345
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	346 If 'leftside' is True, left side of string 's' is trimmed.
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	347 'ellipsis' is always placed at trimmed side.
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	348
34151 414a3513c2bd doctest: do not embed non-ascii characters in docstring Yuya Nishihara <yuya@tcha.org> parents: 34150 diff changeset	349 >>> from .node import bin
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	350 >>> def bprint(s):
a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	351 ... print(pycompat.sysstr(s))
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	352 >>> ellipsis = b'+++'
27355 b479fc425a81 encoding: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26963 diff changeset	353 >>> from . import encoding
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	354 >>> encoding.encoding = b'utf-8'
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	355 >>> t = b'1234567890'
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	356 >>> bprint(trim(t, 12, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	357 1234567890
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	358 >>> bprint(trim(t, 10, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	359 1234567890
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	360 >>> bprint(trim(t, 8, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	361 12345+++
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	362 >>> bprint(trim(t, 8, ellipsis=ellipsis, leftside=True))
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	363 +++67890
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	364 >>> bprint(trim(t, 8))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	365 12345678
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	366 >>> bprint(trim(t, 8, leftside=True))
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	367 34567890
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	368 >>> bprint(trim(t, 3, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	369 +++
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	370 >>> bprint(trim(t, 1, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	371 +
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	372 >>> u = u'\u3042\u3044\u3046\u3048\u304a' # 2 x 5 = 10 columns
34150 e9e225f16932 doctest: pass encoding name as system string Yuya Nishihara <yuya@tcha.org> parents: 34146 diff changeset	373 >>> t = u.encode(pycompat.sysstr(encoding.encoding))
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	374 >>> bprint(trim(t, 12, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	375 \xe3\x81\x82\xe3\x81\x84\xe3\x81\x86\xe3\x81\x88\xe3\x81\x8a
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	376 >>> bprint(trim(t, 10, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	377 \xe3\x81\x82\xe3\x81\x84\xe3\x81\x86\xe3\x81\x88\xe3\x81\x8a
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	378 >>> bprint(trim(t, 8, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	379 \xe3\x81\x82\xe3\x81\x84+++
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	380 >>> bprint(trim(t, 8, ellipsis=ellipsis, leftside=True))
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	381 +++\xe3\x81\x88\xe3\x81\x8a
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	382 >>> bprint(trim(t, 5))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	383 \xe3\x81\x82\xe3\x81\x84
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	384 >>> bprint(trim(t, 5, leftside=True))
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	385 \xe3\x81\x88\xe3\x81\x8a
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	386 >>> bprint(trim(t, 4, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	387 +++
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	388 >>> bprint(trim(t, 4, ellipsis=ellipsis, leftside=True))
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	389 +++
34151 414a3513c2bd doctest: do not embed non-ascii characters in docstring Yuya Nishihara <yuya@tcha.org> parents: 34150 diff changeset	390 >>> t = bin(b'112233445566778899aa') # invalid byte sequence
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	391 >>> bprint(trim(t, 12, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	392 \x11\x22\x33\x44\x55\x66\x77\x88\x99\xaa
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	393 >>> bprint(trim(t, 10, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	394 \x11\x22\x33\x44\x55\x66\x77\x88\x99\xaa
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	395 >>> bprint(trim(t, 8, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	396 \x11\x22\x33\x44\x55+++
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	397 >>> bprint(trim(t, 8, ellipsis=ellipsis, leftside=True))
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	398 +++\x66\x77\x88\x99\xaa
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	399 >>> bprint(trim(t, 8))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	400 \x11\x22\x33\x44\x55\x66\x77\x88
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	401 >>> bprint(trim(t, 8, leftside=True))
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	402 \x33\x44\x55\x66\x77\x88\x99\xaa
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	403 >>> bprint(trim(t, 3, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	404 +++
34152 a8994d08e4a2 doctest: use print_function and convert bytes to unicode where needed Yuya Nishihara <yuya@tcha.org> parents: 34151 diff changeset	405 >>> bprint(trim(t, 1, ellipsis=ellipsis))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	406 +
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	407 """
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	408 try:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	409 u = s.decode(_sysstr(encoding))
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	410 except UnicodeDecodeError:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	411 if len(s) <= width: # trimming is not needed
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	412 return s
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	413 width -= len(ellipsis)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	414 if width <= 0: # no enough room even for ellipsis
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	415 return ellipsis[: width + len(ellipsis)]
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	416 if leftside:
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	417 return ellipsis + s[-width:]
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	418 return s[:width] + ellipsis
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	419
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	420 if ucolwidth(u) <= width: # trimming is not needed
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	421 return s
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	422
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	423 width -= len(ellipsis)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	424 if width <= 0: # no enough room even for ellipsis
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	425 return ellipsis[: width + len(ellipsis)]
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	426
21861 b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	427 if leftside:
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	428 uslice = lambda i: u[i:]
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	429 concat = lambda s: ellipsis + s
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	430 else:
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	431 uslice = lambda i: u[:-i]
b515c3a63e96 encoding: add 'leftside' argument into 'trim' to switch trimming side FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 21856 diff changeset	432 concat = lambda s: s + ellipsis
38823 e7aa113b14f7 global: use pycompat.xrange() Gregory Szorc <gregory.szorc@gmail.com> parents: 38739 diff changeset	433 for i in pycompat.xrange(1, len(u)):
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	434 usub = uslice(i)
d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	435 if ucolwidth(usub) <= width:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	436 return concat(usub.encode(_sysstr(encoding)))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	437 return ellipsis # no enough room for multi-column characters
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	438
21856 d24969ee272f encoding: add 'trim' to trim multi-byte characters at most specified columns FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 18378 diff changeset	439
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	440 def lower(s):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	441 # type: (bytes) -> bytes
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	442 b"best-effort encoding-aware case-folding of local string s"
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	443 try:
22779 d9585dda63c3 encoding.lower: use fast ASCII lower Siddharth Agarwal <sid0@fb.com> parents: 22778 diff changeset	444 return asciilower(s)
17235 3745ae495ce5 encoding: use s.decode to trigger UnicodeDecodeError Martin Geisler <mg@aragost.com> parents: 16493 diff changeset	445 except UnicodeDecodeError:
16387 c481761033bd encoding: add fast-path for ASCII lowercase Matt Mackall <mpm@selenic.com> parents: 16274 diff changeset	446 pass
c481761033bd encoding: add fast-path for ASCII lowercase Matt Mackall <mpm@selenic.com> parents: 16274 diff changeset	447 try:
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	448 if isinstance(s, localstr):
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	449 u = s._utf8.decode("utf-8")
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	450 else:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	451 u = s.decode(_sysstr(encoding), _sysstr(encodingmode))
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	452
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	453 lu = u.lower()
e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	454 if u == lu:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	455 return s # preserve localstring
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	456 return lu.encode(_sysstr(encoding))
14069 e38846a79a23 encoding: add an encoding-aware lower function Matt Mackall <mpm@selenic.com> parents: 13940 diff changeset	457 except UnicodeError:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	458 return s.lower() # we don't know how to fold this except in ASCII
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	459 except LookupError as k:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	460 raise error.Abort(k, hint=b"please check your locale settings")
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	461
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	462
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	463 def upper(s):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	464 # type: (bytes) -> bytes
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	465 b"best-effort encoding-aware case-folding of local string s"
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	466 try:
24578 ac08de78de7f encoding: use parsers.asciiupper when available Siddharth Agarwal <sid0@fb.com> parents: 23596 diff changeset	467 return asciiupper(s)
17236 9fb8312dbdbd encoding: add fast-path for ASCII uppercase. Martin Geisler <mg@aragost.com> parents: 17235 diff changeset	468 except UnicodeDecodeError:
24597 b4258d5a1600 encoding.upper: factor out fallback code Siddharth Agarwal <sid0@fb.com> parents: 24593 diff changeset	469 return upperfallback(s)
b4258d5a1600 encoding.upper: factor out fallback code Siddharth Agarwal <sid0@fb.com> parents: 24593 diff changeset	470
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	471
24597 b4258d5a1600 encoding.upper: factor out fallback code Siddharth Agarwal <sid0@fb.com> parents: 24593 diff changeset	472 def upperfallback(s):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	473 # type: (Any) -> Any
17236 9fb8312dbdbd encoding: add fast-path for ASCII uppercase. Martin Geisler <mg@aragost.com> parents: 17235 diff changeset	474 try:
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	475 if isinstance(s, localstr):
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	476 u = s._utf8.decode("utf-8")
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	477 else:
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	478 u = s.decode(_sysstr(encoding), _sysstr(encodingmode))
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	479
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	480 uu = u.upper()
2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	481 if u == uu:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	482 return s # preserve localstring
30033 02dbfaa6df0b py3: convert encoding name and mode to str Yuya Nishihara <yuya@tcha.org> parents: 30031 diff changeset	483 return uu.encode(_sysstr(encoding))
15672 2ebe3d0ce91d i18n: use encoding.lower/upper for encoding aware case folding FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 15143 diff changeset	484 except UnicodeError:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	485 return s.upper() # we don't know how to fold this except in ASCII
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 24608 diff changeset	486 except LookupError as k:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	487 raise error.Abort(k, hint=b"please check your locale settings")
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	488
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	489
24593 f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	490 class normcasespecs(object):
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	491 '''what a platform's normcase does to ASCII strings
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	492
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	493 This is specified per platform, and should be consistent with what normcase
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	494 on that platform actually does.
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	495
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	496 lower: normcase lowercases ASCII strings
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	497 upper: normcase uppercases ASCII strings
24608 1c533e23ce95 util.h: define an enum for normcase specs Siddharth Agarwal <sid0@fb.com> parents: 24597 diff changeset	498 other: the fallback function should always be called
1c533e23ce95 util.h: define an enum for normcase specs Siddharth Agarwal <sid0@fb.com> parents: 24597 diff changeset	499
1c533e23ce95 util.h: define an enum for normcase specs Siddharth Agarwal <sid0@fb.com> parents: 24597 diff changeset	500 This should be kept in sync with normcase_spec in util.h.'''
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	501
24593 f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	502 lower = -1
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	503 upper = 1
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	504 other = 0
f473a1fe5c7c encoding: define an enum that specifies what normcase does to ASCII strings Siddharth Agarwal <sid0@fb.com> parents: 24578 diff changeset	505
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	506
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	507 def jsonescape(s, paranoid=False):
43544 2ade00f3b03b encoding: add comment-based type hints for pytype Augie Fackler <augie@google.com> parents: 43517 diff changeset	508 # type: (Any, Any) -> Any
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	509 '''returns a string suitable for JSON
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	510
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	511 JSON is problematic for us because it doesn't support non-Unicode
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	512 bytes. To deal with this, we take the following approach:
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	513
37991 3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	514 - localstr/safelocalstr objects are converted back to UTF-8
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	515 - valid UTF-8/ASCII strings are passed as-is
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	516 - other strings are converted to UTF-8b surrogate encoding
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	517 - apply JSON-specified string escaping
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	518
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	519 (escapes are doubled in these tests)
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	520
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	521 >>> jsonescape(b'this is a test')
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	522 'this is a test'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	523 >>> jsonescape(b'escape characters: \\0 \\x0b \\x7f')
27881 ffa599f3f503 encoding: escape U+007F (DEL) character in JSON Yuya Nishihara <yuya@tcha.org> parents: 27699 diff changeset	524 'escape characters: \\\\u0000 \\\\u000b \\\\u007f'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	525 >>> jsonescape(b'escape characters: \\b \\t \\n \\f \\r \\" \\\\')
33943 2c37f9dabc32 encoding: add fast path of jsonescape() (issue5533) Yuya Nishihara <yuya@tcha.org> parents: 33942 diff changeset	526 'escape characters: \\\\b \\\\t \\\\n \\\\f \\\\r \\\\" \\\\\\\\'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	527 >>> jsonescape(b'a weird byte: \\xdd')
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	528 'a weird byte: \\xed\\xb3\\x9d'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	529 >>> jsonescape(b'utf-8: caf\\xc3\\xa9')
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	530 'utf-8: caf\\xc3\\xa9'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	531 >>> jsonescape(b'')
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	532 ''
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	533
28069 b2d24c2898f9 encoding: backport paranoid escaping from templatefilters.jsonescape() Yuya Nishihara <yuya@tcha.org> parents: 28068 diff changeset	534 If paranoid, non-ascii and common troublesome characters are also escaped.
b2d24c2898f9 encoding: backport paranoid escaping from templatefilters.jsonescape() Yuya Nishihara <yuya@tcha.org> parents: 28068 diff changeset	535 This is suitable for web output.
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	536
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	537 >>> s = b'escape characters: \\0 \\x0b \\x7f'
33943 2c37f9dabc32 encoding: add fast path of jsonescape() (issue5533) Yuya Nishihara <yuya@tcha.org> parents: 33942 diff changeset	538 >>> assert jsonescape(s) == jsonescape(s, paranoid=True)
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	539 >>> s = b'escape characters: \\b \\t \\n \\f \\r \\" \\\\'
33943 2c37f9dabc32 encoding: add fast path of jsonescape() (issue5533) Yuya Nishihara <yuya@tcha.org> parents: 33942 diff changeset	540 >>> assert jsonescape(s) == jsonescape(s, paranoid=True)
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	541 >>> jsonescape(b'escape boundary: \\x7e \\x7f \\xc2\\x80', paranoid=True)
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	542 'escape boundary: ~ \\\\u007f \\\\u0080'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	543 >>> jsonescape(b'a weird byte: \\xdd', paranoid=True)
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	544 'a weird byte: \\\\udcdd'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	545 >>> jsonescape(b'utf-8: caf\\xc3\\xa9', paranoid=True)
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	546 'utf-8: caf\\\\u00e9'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	547 >>> jsonescape(b'non-BMP: \\xf0\\x9d\\x84\\x9e', paranoid=True)
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	548 'non-BMP: \\\\ud834\\\\udd1e'
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	549 >>> jsonescape(b'<foo@example.org>', paranoid=True)
28069 b2d24c2898f9 encoding: backport paranoid escaping from templatefilters.jsonescape() Yuya Nishihara <yuya@tcha.org> parents: 28068 diff changeset	550 '\\\\u003cfoo@example.org\\\\u003e'
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	551 '''
f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	552
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	553 u8chars = toutf8b(s)
9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	554 try:
33942 b9101467d88b encoding: extract stub for fast JSON escape Yuya Nishihara <yuya@tcha.org> parents: 33873 diff changeset	555 return _jsonescapeu8fast(u8chars, paranoid)
b9101467d88b encoding: extract stub for fast JSON escape Yuya Nishihara <yuya@tcha.org> parents: 33873 diff changeset	556 except ValueError:
28068 9ece901f7a19 encoding: add option to escape non-ascii characters in JSON Yuya Nishihara <yuya@tcha.org> parents: 28067 diff changeset	557 pass
33942 b9101467d88b encoding: extract stub for fast JSON escape Yuya Nishihara <yuya@tcha.org> parents: 33873 diff changeset	558 return charencodepure.jsonescapeu8fallback(u8chars, paranoid)
22426 f6b533e64ed6 encoding: add json escaping filter Matt Mackall <mpm@selenic.com> parents: 22425 diff changeset	559
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	560
34225 aa877860d4d7 py3: use 'surrogatepass' error handler to process U+DCxx transparently Yuya Nishihara <yuya@tcha.org> parents: 34223 diff changeset	561 # We need to decode/encode U+DCxx codes transparently since invalid UTF-8
aa877860d4d7 py3: use 'surrogatepass' error handler to process U+DCxx transparently Yuya Nishihara <yuya@tcha.org> parents: 34223 diff changeset	562 # bytes are mapped to that range.
aa877860d4d7 py3: use 'surrogatepass' error handler to process U+DCxx transparently Yuya Nishihara <yuya@tcha.org> parents: 34223 diff changeset	563 if pycompat.ispy3:
aa877860d4d7 py3: use 'surrogatepass' error handler to process U+DCxx transparently Yuya Nishihara <yuya@tcha.org> parents: 34223 diff changeset	564 _utf8strict = r'surrogatepass'
aa877860d4d7 py3: use 'surrogatepass' error handler to process U+DCxx transparently Yuya Nishihara <yuya@tcha.org> parents: 34223 diff changeset	565 else:
aa877860d4d7 py3: use 'surrogatepass' error handler to process U+DCxx transparently Yuya Nishihara <yuya@tcha.org> parents: 34223 diff changeset	566 _utf8strict = r'strict'
aa877860d4d7 py3: use 'surrogatepass' error handler to process U+DCxx transparently Yuya Nishihara <yuya@tcha.org> parents: 34223 diff changeset	567
26875 cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	568 _utf8len = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 4]
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	569
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	570
26875 cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	571 def getutf8char(s, pos):
43722 83a349aaeba3 typing: constrain argument/return types of encoding.toutf8b() Yuya Nishihara <yuya@tcha.org> parents: 43721 diff changeset	572 # type: (bytes, int) -> bytes
26875 cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	573 '''get the next full utf-8 character in the given string, starting at pos
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	574
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	575 Raises a UnicodeError if the given location does not start a valid
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	576 utf-8 character.
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	577 '''
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	578
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	579 # find how many bytes to attempt decoding from first nibble
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	580 l = _utf8len[ord(s[pos : pos + 1]) >> 4]
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	581 if not l: # ascii
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	582 return s[pos : pos + 1]
26875 cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	583
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	584 c = s[pos : pos + l]
26875 cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	585 # validate with attempted decode
34225 aa877860d4d7 py3: use 'surrogatepass' error handler to process U+DCxx transparently Yuya Nishihara <yuya@tcha.org> parents: 34223 diff changeset	586 c.decode("utf-8", _utf8strict)
26875 cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	587 return c
cf47bdb2183c encoding: add getutf8char helper Matt Mackall <mpm@selenic.com> parents: 25660 diff changeset	588
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	589
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	590 def toutf8b(s):
43722 83a349aaeba3 typing: constrain argument/return types of encoding.toutf8b() Yuya Nishihara <yuya@tcha.org> parents: 43721 diff changeset	591 # type: (bytes) -> bytes
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	592 '''convert a local, possibly-binary string into UTF-8b
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	593
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	594 This is intended as a generic method to preserve data when working
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	595 with schemes like JSON and XML that have no provision for
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	596 arbitrary byte strings. As Mercurial often doesn't know
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	597 what encoding data is in, we use so-called UTF-8b.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	598
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	599 If a string is already valid UTF-8 (or ASCII), it passes unmodified.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	600 Otherwise, unsupported bytes are mapped to UTF-16 surrogate range,
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	601 uDC00-uDCFF.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	602
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	603 Principles of operation:
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	604
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17236 diff changeset	605 - ASCII and UTF-8 data successfully round-trips and is understood
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	606 by Unicode-oriented clients
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	607 - filenames and file contents in arbitrary other encodings can have
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	608 be round-tripped or recovered by clueful clients
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	609 - local strings that have a cached known UTF-8 encoding (aka
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	610 localstr) get sent as UTF-8 so Unicode-oriented clients get the
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	611 Unicode data they want
37991 3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	612 - non-lossy local strings (aka safelocalstr) get sent as UTF-8 as well
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	613 - because we must preserve UTF-8 bytestring in places such as
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	614 filenames, metadata can't be roundtripped without help
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	615
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	616 (Note: "UTF-8b" often refers to decoding a mix of valid UTF-8 and
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	617 arbitrary bytes into an internal Unicode format that can be
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	618 re-encoded back into the original. Here we are exposing the
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	619 internal surrogate encoding as a UTF-8 string.)
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	620 '''
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	621
37990 57b0c7221dba encoding: fix toutf8b() to resurrect lossy characters even if "\xed" in it Yuya Nishihara <yuya@tcha.org> parents: 36805 diff changeset	622 if isinstance(s, localstr):
57b0c7221dba encoding: fix toutf8b() to resurrect lossy characters even if "\xed" in it Yuya Nishihara <yuya@tcha.org> parents: 36805 diff changeset	623 # assume that the original UTF-8 sequence would never contain
57b0c7221dba encoding: fix toutf8b() to resurrect lossy characters even if "\xed" in it Yuya Nishihara <yuya@tcha.org> parents: 36805 diff changeset	624 # invalid characters in U+DCxx range
57b0c7221dba encoding: fix toutf8b() to resurrect lossy characters even if "\xed" in it Yuya Nishihara <yuya@tcha.org> parents: 36805 diff changeset	625 return s._utf8
37991 3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	626 elif isinstance(s, safelocalstr):
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	627 # already verified that s is non-lossy in legacy encoding, which
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	628 # shouldn't contain characters in U+DCxx range
3ea3c96ada54 encoding: introduce tagging type for non-lossy non-ASCII string Yuya Nishihara <yuya@tcha.org> parents: 37990 diff changeset	629 return fromlocal(s)
37990 57b0c7221dba encoding: fix toutf8b() to resurrect lossy characters even if "\xed" in it Yuya Nishihara <yuya@tcha.org> parents: 36805 diff changeset	630 elif isasciistr(s):
33946 6c119dbfd0c0 encoding: add fast path of from/toutf8b() for ASCII strings Yuya Nishihara <yuya@tcha.org> parents: 33945 diff changeset	631 return s
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	632 if b"\xed" not in s:
26879 a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	633 try:
34225 aa877860d4d7 py3: use 'surrogatepass' error handler to process U+DCxx transparently Yuya Nishihara <yuya@tcha.org> parents: 34223 diff changeset	634 s.decode('utf-8', _utf8strict)
26879 a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	635 return s
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	636 except UnicodeDecodeError:
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	637 pass
26878 d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	638
34223 1c601df9894c py3: wrap bytes in encoding.from/toutf8b() with bytestr Yuya Nishihara <yuya@tcha.org> parents: 34207 diff changeset	639 s = pycompat.bytestr(s)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	640 r = b""
26878 d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	641 pos = 0
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	642 l = len(s)
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	643 while pos < l:
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	644 try:
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	645 c = getutf8char(s, pos)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	646 if b"\xed\xb0\x80" <= c <= b"\xed\xb3\xbf":
26879 a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	647 # have to re-escape existing U+DCxx characters
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	648 c = unichr(0xDC00 + ord(s[pos])).encode('utf-8', _utf8strict)
26879 a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	649 pos += 1
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	650 else:
a24b98f4e03c encoding: re-escape U+DCxx characters in toutf8b input (issue4927) Matt Mackall <mpm@selenic.com> parents: 26878 diff changeset	651 pos += len(c)
26878 d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	652 except UnicodeDecodeError:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	653 c = unichr(0xDC00 + ord(s[pos])).encode('utf-8', _utf8strict)
26878 d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	654 pos += 1
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	655 r += c
d7e83f106459 encoding: use getutf8char in toutf8b Matt Mackall <mpm@selenic.com> parents: 26877 diff changeset	656 return r
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	657
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	658
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	659 def fromutf8b(s):
43721 b65fcccd9100 typing: fix argument type of encoding.tolocal() and .fromutf8b() Yuya Nishihara <yuya@tcha.org> parents: 43720 diff changeset	660 # type: (bytes) -> bytes
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	661 '''Given a UTF-8b string, return a local, possibly-binary string.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	662
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	663 return the original binary string. This
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	664 is a round-trip process for strings like filenames, but metadata
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	665 that's was passed through tolocal will remain in UTF-8.
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	666
26963 de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	667 >>> roundtrip = lambda x: fromutf8b(toutf8b(x)) == x
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	668 >>> m = b"\\xc3\\xa9\\x99abcd"
26963 de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	669 >>> toutf8b(m)
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	670 '\\xc3\\xa9\\xed\\xb2\\x99abcd'
26963 de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	671 >>> roundtrip(m)
de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	672 True
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	673 >>> roundtrip(b"\\xc2\\xc2\\x80")
26963 de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	674 True
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	675 >>> roundtrip(b"\\xef\\xbf\\xbd")
26963 de5ae97ce9f4 encoding: extend test cases for utf8b Matt Mackall <mpm@selenic.com> parents: 26879 diff changeset	676 True
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	677 >>> roundtrip(b"\\xef\\xef\\xbf\\xbd")
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	678 True
34146 0fa781320203 doctest: bulk-replace string literals with b'' for Python 3 Yuya Nishihara <yuya@tcha.org> parents: 33946 diff changeset	679 >>> roundtrip(b"\\xf1\\x80\\x80\\x80\\x80")
27699 c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	680 True
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	681 '''
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	682
33946 6c119dbfd0c0 encoding: add fast path of from/toutf8b() for ASCII strings Yuya Nishihara <yuya@tcha.org> parents: 33945 diff changeset	683 if isasciistr(s):
6c119dbfd0c0 encoding: add fast path of from/toutf8b() for ASCII strings Yuya Nishihara <yuya@tcha.org> parents: 33945 diff changeset	684 return s
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	685 # fast path - look for uDxxx prefixes in s
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	686 if b"\xed" not in s:
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	687 return s
84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	688
27699 c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	689 # We could do this with the unicode type but some Python builds
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	690 # use UTF-16 internally (issue5031) which causes non-BMP code
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	691 # points to be escaped. Instead, we use our handy getutf8char
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	692 # helper again to walk the string without "decoding" it.
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	693
34223 1c601df9894c py3: wrap bytes in encoding.from/toutf8b() with bytestr Yuya Nishihara <yuya@tcha.org> parents: 34207 diff changeset	694 s = pycompat.bytestr(s)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	695 r = b""
27699 c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	696 pos = 0
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	697 l = len(s)
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	698 while pos < l:
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	699 c = getutf8char(s, pos)
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	700 pos += len(c)
c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	701 # unescape U+DCxx characters
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	702 if b"\xed\xb0\x80" <= c <= b"\xed\xb3\xbf":
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41841 diff changeset	703 c = pycompat.bytechr(ord(c.decode("utf-8", _utf8strict)) & 0xFF)
27699 c8d3392f76e1 encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Matt Mackall <mpm@selenic.com> parents: 27356 diff changeset	704 r += c
16133 84c58da3a1f8 encoding: introduce utf8-b helpers Matt Mackall <mpm@selenic.com> parents: 15769 diff changeset	705 return r

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/encoding.py @ 43725:da925257a39e