mercurial-scm/hg: mercurial/utils/cborutil.py annotate

annotate mercurial/utils/cborutil.py @ 52644:e627cc25b6f3

pyupgrade: rewrite `yield` statements in a loop to `yield from` This is the `legacy` fixer in `pyupgrade`, with the `yield` statement yielding loop commented back in. This seems to help pytype in some cases, and hurt it in others. But that can be manually fixed later. Note that it's possibly buggy in that it aggressively changed `import-checker.py` to `yield from 'fcntl', 'grp', 'pwd', 'select', 'termios': # Unix only`, which is invalid syntax. Possibly it needed help from the token fixer that I've disabled locally (because that wants to make a bunch of unrelated changes). Just change those few places to yield from a list, to avoid having to constantly revert that.

author	Matt Harbison <matt_harbison@yahoo.com>
date	Sun, 05 Jan 2025 22:26:16 -0500
parents	fb8932685031
children	279e217d6041

rev	line source
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # cborutil.py - CBOR extensions
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
51859 f4733654f144 typing: add `from __future__ import annotations` to most files Matt Harbison <matt_harbison@yahoo.com> parents: 48946 diff changeset	8 from __future__ import annotations
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	10 import struct
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	11
39456 8d858fbf2759 cbor: teach the encoder to handle python `long` type for Windows Matt Harbison <matt_harbison@yahoo.com> parents: 39413 diff changeset	12
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	13 # Very short very of RFC 7049...
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	14 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	15 # Each item begins with a byte. The 3 high bits of that byte denote the
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	16 # "major type." The lower 5 bits denote the "subtype." Each major type
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	17 # has its own encoding mechanism.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	18 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	19 # Most types have lengths. However, bytestring, string, array, and map
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	20 # can be indefinite length. These are denotes by a subtype with value 31.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	21 # Sub-components of those types then come afterwards and are terminated
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	22 # by a "break" byte.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	23
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	24 MAJOR_TYPE_UINT = 0
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	25 MAJOR_TYPE_NEGINT = 1
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	26 MAJOR_TYPE_BYTESTRING = 2
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	27 MAJOR_TYPE_STRING = 3
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	28 MAJOR_TYPE_ARRAY = 4
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29 MAJOR_TYPE_MAP = 5
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30 MAJOR_TYPE_SEMANTIC = 6
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31 MAJOR_TYPE_SPECIAL = 7
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	33 SUBTYPE_MASK = 0b00011111
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	34
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	35 SUBTYPE_FALSE = 20
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	36 SUBTYPE_TRUE = 21
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	37 SUBTYPE_NULL = 22
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	38 SUBTYPE_HALF_FLOAT = 25
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	39 SUBTYPE_SINGLE_FLOAT = 26
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	40 SUBTYPE_DOUBLE_FLOAT = 27
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	41 SUBTYPE_INDEFINITE = 31
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	42
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	43 SEMANTIC_TAG_FINITE_SET = 258
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	44
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	45 # Indefinite types begin with their major type ORd with information value 31.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	46 BEGIN_INDEFINITE_BYTESTRING = struct.pack(
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	47 '>B', MAJOR_TYPE_BYTESTRING << 5 \| SUBTYPE_INDEFINITE
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	48 )
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	49 BEGIN_INDEFINITE_ARRAY = struct.pack(
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	50 '>B', MAJOR_TYPE_ARRAY << 5 \| SUBTYPE_INDEFINITE
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	51 )
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	52 BEGIN_INDEFINITE_MAP = struct.pack(
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	53 '>B', MAJOR_TYPE_MAP << 5 \| SUBTYPE_INDEFINITE
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	54 )
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	55
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	56 ENCODED_LENGTH_1 = struct.Struct('>B')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	57 ENCODED_LENGTH_2 = struct.Struct('>BB')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	58 ENCODED_LENGTH_3 = struct.Struct('>BH')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	59 ENCODED_LENGTH_4 = struct.Struct('>BL')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	60 ENCODED_LENGTH_5 = struct.Struct('>BQ')
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	61
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	62 # The break ends an indefinite length item.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	63 BREAK = b'\xff'
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	64 BREAK_INT = 255
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	65
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	66
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	67 def encodelength(majortype, length):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	68 """Obtain a value encoding the major type and its length."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	69 if length < 24:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	70 return ENCODED_LENGTH_1.pack(majortype << 5 \| length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	71 elif length < 256:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	72 return ENCODED_LENGTH_2.pack(majortype << 5 \| 24, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	73 elif length < 65536:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	74 return ENCODED_LENGTH_3.pack(majortype << 5 \| 25, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	75 elif length < 4294967296:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	76 return ENCODED_LENGTH_4.pack(majortype << 5 \| 26, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	77 else:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	78 return ENCODED_LENGTH_5.pack(majortype << 5 \| 27, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	79
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	80
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	81 def streamencodebytestring(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	82 yield encodelength(MAJOR_TYPE_BYTESTRING, len(v))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	83 yield v
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	84
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	85
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	86 def streamencodebytestringfromiter(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	87 """Convert an iterator of chunks to an indefinite bytestring.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	88
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	89 Given an input that is iterable and each element in the iterator is
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	90 representable as bytes, emit an indefinite length bytestring.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	91 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	92 yield BEGIN_INDEFINITE_BYTESTRING
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	93
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	94 for chunk in it:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	95 yield encodelength(MAJOR_TYPE_BYTESTRING, len(chunk))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	96 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	97
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	98 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	99
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	100
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	101 def streamencodeindefinitebytestring(source, chunksize=65536):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	102 """Given a large source buffer, emit as an indefinite length bytestring.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	103
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	104 This is a generator of chunks constituting the encoded CBOR data.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	105 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	106 yield BEGIN_INDEFINITE_BYTESTRING
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	107
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	108 i = 0
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	109 l = len(source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	110
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	111 while True:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	112 chunk = source[i : i + chunksize]
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	113 i += len(chunk)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	114
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	115 yield encodelength(MAJOR_TYPE_BYTESTRING, len(chunk))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	116 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	117
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	118 if i >= l:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	119 break
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	120
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	121 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	122
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	123
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	124 def streamencodeint(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	125 if v >= 18446744073709551616 or v < -18446744073709551616:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	126 raise ValueError(b'big integers not supported')
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	127
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	128 if v >= 0:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	129 yield encodelength(MAJOR_TYPE_UINT, v)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	130 else:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	131 yield encodelength(MAJOR_TYPE_NEGINT, abs(v) - 1)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	132
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	133
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	134 def streamencodearray(l):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	135 """Encode a known size iterable to an array."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	136
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	137 yield encodelength(MAJOR_TYPE_ARRAY, len(l))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	138
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	139 for i in l:
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52634 diff changeset	140 yield from streamencode(i)
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	141
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	142
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	143 def streamencodearrayfromiter(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	144 """Encode an iterator of items to an indefinite length array."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	145
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	146 yield BEGIN_INDEFINITE_ARRAY
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	147
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	148 for i in it:
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52634 diff changeset	149 yield from streamencode(i)
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	150
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	151 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	152
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	153
37898 2ae6a3134362 cborutil: port to Python 3 Augie Fackler <augie@google.com> parents: 37711 diff changeset	154 def _mixedtypesortkey(v):
2ae6a3134362 cborutil: port to Python 3 Augie Fackler <augie@google.com> parents: 37711 diff changeset	155 return type(v).__name__, v
2ae6a3134362 cborutil: port to Python 3 Augie Fackler <augie@google.com> parents: 37711 diff changeset	156
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	157
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	158 def streamencodeset(s):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	159 # https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml defines
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	160 # semantic tag 258 for finite sets.
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	161 yield encodelength(MAJOR_TYPE_SEMANTIC, SEMANTIC_TAG_FINITE_SET)
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	162
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52634 diff changeset	163 yield from streamencodearray(sorted(s, key=_mixedtypesortkey))
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	164
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	165
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	166 def streamencodemap(d):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	167 """Encode dictionary to a generator.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	168
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	169 Does not supporting indefinite length dictionaries.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	170 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	171 yield encodelength(MAJOR_TYPE_MAP, len(d))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	172
48913 f254fc73d956 global: bulk replace simple pycompat.iteritems(x) with x.items() Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	173 for key, value in sorted(d.items(), key=lambda x: _mixedtypesortkey(x[0])):
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52634 diff changeset	174 yield from streamencode(key)
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52634 diff changeset	175 yield from streamencode(value)
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	176
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	177
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	178 def streamencodemapfromiter(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	179 """Given an iterable of (key, value), encode to an indefinite length map."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	180 yield BEGIN_INDEFINITE_MAP
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	181
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	182 for key, value in it:
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52634 diff changeset	183 yield from streamencode(key)
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52634 diff changeset	184 yield from streamencode(value)
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	185
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	186 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	187
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	188
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	189 def streamencodebool(b):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	190 # major type 7, simple value 20 and 21.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	191 yield b'\xf5' if b else b'\xf4'
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	192
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	193
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	194 def streamencodenone(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	195 # major type 7, simple value 22.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	196 yield b'\xf6'
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	197
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	198
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	199 STREAM_ENCODERS = {
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	200 bytes: streamencodebytestring,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	201 int: streamencodeint,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	202 list: streamencodearray,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	203 tuple: streamencodearray,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	204 dict: streamencodemap,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	205 set: streamencodeset,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	206 bool: streamencodebool,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	207 type(None): streamencodenone,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	208 }
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	209
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	210
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	211 def streamencode(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	212 """Encode a value in a streaming manner.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	213
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	214 Given an input object, encode it to CBOR recursively.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	215
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	216 Returns a generator of CBOR encoded bytes. There is no guarantee
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	217 that each emitted chunk fully decodes to a value or sub-value.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	218
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	219 Encoding is deterministic - unordered collections are sorted.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	220 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	221 fn = STREAM_ENCODERS.get(v.__class__)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	222
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	223 if not fn:
42480 b6387a65851d cborutil: fix streamencode() to handle subtypes Yuya Nishihara <yuya@tcha.org> parents: 40124 diff changeset	224 # handle subtypes such as encoding.localstr and util.sortdict
b6387a65851d cborutil: fix streamencode() to handle subtypes Yuya Nishihara <yuya@tcha.org> parents: 40124 diff changeset	225 for ty in STREAM_ENCODERS:
b6387a65851d cborutil: fix streamencode() to handle subtypes Yuya Nishihara <yuya@tcha.org> parents: 40124 diff changeset	226 if not isinstance(v, ty):
b6387a65851d cborutil: fix streamencode() to handle subtypes Yuya Nishihara <yuya@tcha.org> parents: 40124 diff changeset	227 continue
b6387a65851d cborutil: fix streamencode() to handle subtypes Yuya Nishihara <yuya@tcha.org> parents: 40124 diff changeset	228 fn = STREAM_ENCODERS[ty]
b6387a65851d cborutil: fix streamencode() to handle subtypes Yuya Nishihara <yuya@tcha.org> parents: 40124 diff changeset	229 break
b6387a65851d cborutil: fix streamencode() to handle subtypes Yuya Nishihara <yuya@tcha.org> parents: 40124 diff changeset	230
b6387a65851d cborutil: fix streamencode() to handle subtypes Yuya Nishihara <yuya@tcha.org> parents: 40124 diff changeset	231 if not fn:
52634 fb8932685031 cborutil: fix an error interpolating str into bytes Matt Harbison <matt_harbison@yahoo.com> parents: 51874 diff changeset	232 raise ValueError('do not know how to encode %s' % type(v))
37711 65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	233
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	234 return fn(v)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	235
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	236
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	237 class CBORDecodeError(Exception):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	238 """Represents an error decoding CBOR."""
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	239
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	240
48942 bce8f66d3045 cborutil: remove Python 2 definition of _elementtointeger() Gregory Szorc <gregory.szorc@gmail.com> parents: 48932 diff changeset	241 def _elementtointeger(b, i):
bce8f66d3045 cborutil: remove Python 2 definition of _elementtointeger() Gregory Szorc <gregory.szorc@gmail.com> parents: 48932 diff changeset	242 return b[i]
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	243
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	244
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	245 STRUCT_BIG_UBYTE = struct.Struct('>B')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	246 STRUCT_BIG_USHORT = struct.Struct(b'>H')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	247 STRUCT_BIG_ULONG = struct.Struct(b'>L')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	248 STRUCT_BIG_ULONGLONG = struct.Struct(b'>Q')
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	249
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	250 SPECIAL_NONE = 0
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	251 SPECIAL_START_INDEFINITE_BYTESTRING = 1
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	252 SPECIAL_START_ARRAY = 2
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	253 SPECIAL_START_MAP = 3
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	254 SPECIAL_START_SET = 4
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	255 SPECIAL_INDEFINITE_BREAK = 5
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	256
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	257
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	258 def decodeitem(b, offset=0):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	259 """Decode a new CBOR value from a buffer at offset.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	260
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	261 This function attempts to decode up to one complete CBOR value
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	262 from ``b`` starting at offset ``offset``.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	263
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	264 The beginning of a collection (such as an array, map, set, or
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	265 indefinite length bytestring) counts as a single value. For these
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	266 special cases, a state flag will indicate that a special value was seen.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	267
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	268 When called, the function either returns a decoded value or gives
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	269 a hint as to how many more bytes are needed to do so. By calling
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	270 the function repeatedly given a stream of bytes, the caller can
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	271 build up the original values.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	272
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	273 Returns a tuple with the following elements:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	274
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	275 * Bool indicating whether a complete value was decoded.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	276 * A decoded value if first value is True otherwise None
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	277 * Integer number of bytes. If positive, the number of bytes
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	278 read. If negative, the number of bytes we need to read to
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	279 decode this value or the next chunk in this value.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	280 * One of the ``SPECIAL_*`` constants indicating special treatment
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	281 for this value. ``SPECIAL_NONE`` means this is a fully decoded
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	282 simple value (such as an integer or bool).
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	283 """
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	284
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	285 initial = _elementtointeger(b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	286 offset += 1
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	287
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	288 majortype = initial >> 5
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	289 subtype = initial & SUBTYPE_MASK
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	290
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	291 if majortype == MAJOR_TYPE_UINT:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	292 complete, value, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	293
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	294 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	295 return True, value, readcount + 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	296 else:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	297 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	298
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	299 elif majortype == MAJOR_TYPE_NEGINT:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	300 # Negative integers are the same as UINT except inverted minus 1.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	301 complete, value, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	302
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	303 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	304 return True, -value - 1, readcount + 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	305 else:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	306 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	307
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	308 elif majortype == MAJOR_TYPE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	309 # Beginning of bytestrings are treated as uints in order to
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	310 # decode their length, which may be indefinite.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	311 complete, size, readcount = decodeuint(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	312 subtype, b, offset, allowindefinite=True
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	313 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	314
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	315 # We don't know the size of the bytestring. It must be a definitive
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	316 # length since the indefinite subtype would be encoded in the initial
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	317 # byte.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	318 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	319 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	320
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	321 # We know the length of the bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	322 if size is not None:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	323 # And the data is available in the buffer.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	324 if offset + readcount + size <= len(b):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	325 value = b[offset + readcount : offset + readcount + size]
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	326 return True, value, readcount + size + 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	327
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	328 # And we need more data in order to return the bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	329 else:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	330 wanted = len(b) - offset - readcount - size
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	331 return False, None, wanted, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	332
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	333 # It is an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	334 else:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	335 return True, None, 1, SPECIAL_START_INDEFINITE_BYTESTRING
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	336
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	337 elif majortype == MAJOR_TYPE_STRING:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	338 raise CBORDecodeError(b'string major type not supported')
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	339
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	340 elif majortype == MAJOR_TYPE_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	341 # Beginning of arrays are treated as uints in order to decode their
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	342 # length. We don't allow indefinite length arrays.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	343 complete, size, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	344
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	345 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	346 return True, size, readcount + 1, SPECIAL_START_ARRAY
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	347 else:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	348 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	349
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	350 elif majortype == MAJOR_TYPE_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	351 # Beginning of maps are treated as uints in order to decode their
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	352 # number of elements. We don't allow indefinite length arrays.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	353 complete, size, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	354
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	355 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	356 return True, size, readcount + 1, SPECIAL_START_MAP
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	357 else:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	358 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	359
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	360 elif majortype == MAJOR_TYPE_SEMANTIC:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	361 # Semantic tag value is read the same as a uint.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	362 complete, tagvalue, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	363
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	364 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	365 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	366
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	367 # This behavior here is a little wonky. The main type being "decorated"
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	368 # by this semantic tag follows. A more robust parser would probably emit
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	369 # a special flag indicating this as a semantic tag and let the caller
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	370 # deal with the types that follow. But since we don't support many
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	371 # semantic tags, it is easier to deal with the special cases here and
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	372 # hide complexity from the caller. If we add support for more semantic
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	373 # tags, we should probably move semantic tag handling into the caller.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	374 if tagvalue == SEMANTIC_TAG_FINITE_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	375 if offset + readcount >= len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	376 return False, None, -1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	377
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	378 complete, size, readcount2, special = decodeitem(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	379 b, offset + readcount
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	380 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	381
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	382 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	383 return False, None, readcount2, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	384
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	385 if special != SPECIAL_START_ARRAY:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	386 raise CBORDecodeError(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43106 diff changeset	387 b'expected array after finite set semantic tag'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	388 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	389
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	390 return True, size, readcount + readcount2 + 1, SPECIAL_START_SET
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	391
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	392 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	393 raise CBORDecodeError(b'semantic tag %d not allowed' % tagvalue)
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	394
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	395 elif majortype == MAJOR_TYPE_SPECIAL:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	396 # Only specific values for the information field are allowed.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	397 if subtype == SUBTYPE_FALSE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	398 return True, False, 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	399 elif subtype == SUBTYPE_TRUE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	400 return True, True, 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	401 elif subtype == SUBTYPE_NULL:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	402 return True, None, 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	403 elif subtype == SUBTYPE_INDEFINITE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	404 return True, None, 1, SPECIAL_INDEFINITE_BREAK
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	405 # If value is 24, subtype is in next byte.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	406 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	407 raise CBORDecodeError(b'special type %d not allowed' % subtype)
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	408 else:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	409 assert False
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	410
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	411
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	412 def decodeuint(subtype, b, offset=0, allowindefinite=False):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	413 """Decode an unsigned integer.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	414
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	415 ``subtype`` is the lower 5 bits from the initial byte CBOR item
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	416 "header." ``b`` is a buffer containing bytes. ``offset`` points to
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	417 the index of the first byte after the byte that ``subtype`` was
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	418 derived from.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	419
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	420 ``allowindefinite`` allows the special indefinite length value
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	421 indicator.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	422
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	423 Returns a 3-tuple of (successful, value, count).
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	424
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	425 The first element is a bool indicating if decoding completed. The 2nd
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	426 is the decoded integer value or None if not fully decoded or the subtype
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	427 is 31 and ``allowindefinite`` is True. The 3rd value is the count of bytes.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	428 If positive, it is the number of additional bytes decoded. If negative,
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	429 it is the number of additional bytes needed to decode this value.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	430 """
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	431
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	432 # Small values are inline.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	433 if subtype < 24:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	434 return True, subtype, 0
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	435 # Indefinite length specifier.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	436 elif subtype == 31:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	437 if allowindefinite:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	438 return True, None, 0
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	439 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	440 raise CBORDecodeError(b'indefinite length uint not allowed here')
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	441 elif subtype >= 28:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	442 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	443 b'unsupported subtype on integer type: %d' % subtype
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	444 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	445
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	446 if subtype == 24:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	447 s = STRUCT_BIG_UBYTE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	448 elif subtype == 25:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	449 s = STRUCT_BIG_USHORT
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	450 elif subtype == 26:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	451 s = STRUCT_BIG_ULONG
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	452 elif subtype == 27:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	453 s = STRUCT_BIG_ULONGLONG
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	454 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	455 raise CBORDecodeError(b'bounds condition checking violation')
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	456
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	457 if len(b) - offset >= s.size:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	458 return True, s.unpack_from(b, offset)[0], s.size
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	459 else:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	460 return False, None, len(b) - offset - s.size
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	461
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	462
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	463 class bytestringchunk(bytes):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	464 """Represents a chunk/segment in an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	465
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	466 This behaves like a ``bytes`` but in addition has the ``isfirst``
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	467 and ``islast`` attributes indicating whether this chunk is the first
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	468 or last in an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	469 """
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	470
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	471 def __new__(cls, v, first=False, last=False):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	472 self = bytes.__new__(cls, v)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	473 self.isfirst = first
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	474 self.islast = last
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	475
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	476 return self
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	477
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	478
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48942 diff changeset	479 class sansiodecoder:
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	480 """A CBOR decoder that doesn't perform its own I/O.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	481
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	482 To use, construct an instance and feed it segments containing
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	483 CBOR-encoded bytes via ``decode()``. The return value from ``decode()``
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	484 indicates whether a fully-decoded value is available, how many bytes
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	485 were consumed, and offers a hint as to how many bytes should be fed
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	486 in next time to decode the next value.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	487
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	488 The decoder assumes it will decode N discrete CBOR values, not just
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	489 a single value. i.e. if the bytestream contains uints packed one after
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	490 the other, the decoder will decode them all, rather than just the initial
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	491 one.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	492
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	493 When ``decode()`` indicates a value is available, call ``getavailable()``
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	494 to return all fully decoded values.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	495
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	496 ``decode()`` can partially decode input. It is up to the caller to keep
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	497 track of what data was consumed and to pass unconsumed data in on the
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	498 next invocation.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	499
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	500 The decoder decodes atomically at the item level. See ``decodeitem()``.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	501 If an item cannot be fully decoded, the decoder won't record it as
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	502 partially consumed. Instead, the caller will be instructed to pass in
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	503 the initial bytes of this item on the next invocation. This does result
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	504 in some redundant parsing. But the overhead should be minimal.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	505
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	506 This decoder only supports a subset of CBOR as required by Mercurial.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	507 It lacks support for:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	508
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	509 * Indefinite length arrays
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	510 * Indefinite length maps
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	511 * Use of indefinite length bytestrings as keys or values within
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	512 arrays, maps, or sets.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	513 * Nested arrays, maps, or sets within sets
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	514 * Any semantic tag that isn't a mathematical finite set
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	515 * Floating point numbers
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	516 * Undefined special value
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	517
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	518 CBOR types are decoded to Python types as follows:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	519
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	520 uint -> int
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	521 negint -> int
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	522 bytestring -> bytes
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	523 map -> dict
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	524 array -> list
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	525 True -> bool
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	526 False -> bool
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	527 null -> None
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	528 indefinite length bytestring chunk -> [bytestringchunk]
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	529
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	530 The only non-obvious mapping here is an indefinite length bytestring
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	531 to the ``bytestringchunk`` type. This is to facilitate streaming
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	532 indefinite length bytestrings out of the decoder and to differentiate
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	533 a regular bytestring from an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	534 """
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	535
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	536 _STATE_NONE = 0
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	537 _STATE_WANT_MAP_KEY = 1
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	538 _STATE_WANT_MAP_VALUE = 2
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	539 _STATE_WANT_ARRAY_VALUE = 3
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	540 _STATE_WANT_SET_VALUE = 4
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	541 _STATE_WANT_BYTESTRING_CHUNK_FIRST = 5
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	542 _STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT = 6
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	543
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	544 def __init__(self):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	545 # TODO add support for limiting size of bytestrings
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	546 # TODO add support for limiting number of keys / values in collections
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	547 # TODO add support for limiting size of buffered partial values
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	548
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	549 self.decodedbytecount = 0
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	550
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	551 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	552
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	553 # Stack of active nested collections. Each entry is a dict describing
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	554 # the collection.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	555 self._collectionstack = []
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	556
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	557 # Fully decoded key to use for the current map.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	558 self._currentmapkey = None
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	559
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	560 # Fully decoded values available for retrieval.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	561 self._decodedvalues = []
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	562
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	563 @property
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	564 def inprogress(self):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	565 """Whether the decoder has partially decoded a value."""
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	566 return self._state != self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	567
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	568 def decode(self, b, offset=0):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	569 """Attempt to decode bytes from an input buffer.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	570
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	571 ``b`` is a collection of bytes and ``offset`` is the byte
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	572 offset within that buffer from which to begin reading data.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	573
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	574 ``b`` must support ``len()`` and accessing bytes slices via
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	575 ``__slice__``. Typically ``bytes`` instances are used.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	576
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	577 Returns a tuple with the following fields:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	578
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	579 * Bool indicating whether values are available for retrieval.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	580 * Integer indicating the number of bytes that were fully consumed,
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	581 starting from ``offset``.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	582 * Integer indicating the number of bytes that are desired for the
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	583 next call in order to decode an item.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	584 """
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	585 if not b:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	586 return bool(self._decodedvalues), 0, 0
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	587
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	588 initialoffset = offset
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	589
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	590 # We could easily split the body of this loop into a function. But
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	591 # Python performance is sensitive to function calls and collections
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	592 # are composed of many items. So leaving as a while loop could help
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	593 # with performance. One thing that may not help is the use of
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	594 # if..elif versus a lookup/dispatch table. There may be value
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	595 # in switching that.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	596 while offset < len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	597 # Attempt to decode an item. This could be a whole value or a
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	598 # special value indicating an event, such as start or end of a
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	599 # collection or indefinite length type.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	600 complete, value, readcount, special = decodeitem(b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	601
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	602 if readcount > 0:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	603 self.decodedbytecount += readcount
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	604
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	605 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	606 assert readcount < 0
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	607 return (
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	608 bool(self._decodedvalues),
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	609 offset - initialoffset,
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	610 -readcount,
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	611 )
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	612
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	613 offset += readcount
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	614
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	615 # No nested state. We either have a full value or beginning of a
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	616 # complex value to deal with.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	617 if self._state == self._STATE_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	618 # A normal value.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	619 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	620 self._decodedvalues.append(value)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	621
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	622 elif special == SPECIAL_START_ARRAY:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	623 self._collectionstack.append(
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	624 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	625 b'remaining': value,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	626 b'v': [],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	627 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	628 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	629 self._state = self._STATE_WANT_ARRAY_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	630
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	631 elif special == SPECIAL_START_MAP:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	632 self._collectionstack.append(
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	633 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	634 b'remaining': value,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	635 b'v': {},
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	636 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	637 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	638 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	639
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	640 elif special == SPECIAL_START_SET:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	641 self._collectionstack.append(
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	642 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	643 b'remaining': value,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	644 b'v': set(),
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	645 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	646 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	647 self._state = self._STATE_WANT_SET_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	648
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	649 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	650 self._state = self._STATE_WANT_BYTESTRING_CHUNK_FIRST
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	651
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	652 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	653 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	654 b'unhandled special state: %d' % special
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	655 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	656
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	657 # This value becomes an element of the current array.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	658 elif self._state == self._STATE_WANT_ARRAY_VALUE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	659 # Simple values get appended.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	660 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	661 c = self._collectionstack[-1]
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	662 c[b'v'].append(value)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	663 c[b'remaining'] -= 1
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	664
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	665 # self._state doesn't need changed.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	666
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	667 # An array nested within an array.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	668 elif special == SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	669 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	670 newvalue = []
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	671
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	672 lastc[b'v'].append(newvalue)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	673 lastc[b'remaining'] -= 1
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	674
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	675 self._collectionstack.append(
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	676 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	677 b'remaining': value,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	678 b'v': newvalue,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	679 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	680 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	681
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	682 # self._state doesn't need changed.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	683
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	684 # A map nested within an array.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	685 elif special == SPECIAL_START_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	686 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	687 newvalue = {}
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	688
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	689 lastc[b'v'].append(newvalue)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	690 lastc[b'remaining'] -= 1
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	691
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	692 self._collectionstack.append(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	693 {b'remaining': value, b'v': newvalue}
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	694 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	695
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	696 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	697
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	698 elif special == SPECIAL_START_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	699 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	700 newvalue = set()
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	701
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	702 lastc[b'v'].append(newvalue)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	703 lastc[b'remaining'] -= 1
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	704
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	705 self._collectionstack.append(
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	706 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	707 b'remaining': value,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	708 b'v': newvalue,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	709 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	710 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	711
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	712 self._state = self._STATE_WANT_SET_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	713
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	714 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	715 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	716 b'indefinite length bytestrings '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	717 b'not allowed as array values'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	718 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	719
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	720 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	721 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	722 b'unhandled special item when '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	723 b'expecting array value: %d' % special
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	724 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	725
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	726 # This value becomes the key of the current map instance.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	727 elif self._state == self._STATE_WANT_MAP_KEY:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	728 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	729 self._currentmapkey = value
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	730 self._state = self._STATE_WANT_MAP_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	731
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	732 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	733 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	734 b'indefinite length bytestrings '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	735 b'not allowed as map keys'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	736 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	737
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	738 elif special in (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	739 SPECIAL_START_ARRAY,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	740 SPECIAL_START_MAP,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	741 SPECIAL_START_SET,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	742 ):
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	743 raise CBORDecodeError(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43106 diff changeset	744 b'collections not supported as map keys'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	745 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	746
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	747 # We do not allow special values to be used as map keys.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	748 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	749 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	750 b'unhandled special item when '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	751 b'expecting map key: %d' % special
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	752 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	753
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	754 # This value becomes the value of the current map key.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	755 elif self._state == self._STATE_WANT_MAP_VALUE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	756 # Simple values simply get inserted into the map.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	757 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	758 lastc = self._collectionstack[-1]
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	759 lastc[b'v'][self._currentmapkey] = value
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	760 lastc[b'remaining'] -= 1
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	761
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	762 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	763
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	764 # A new array is used as the map value.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	765 elif special == SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	766 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	767 newvalue = []
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	768
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	769 lastc[b'v'][self._currentmapkey] = newvalue
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	770 lastc[b'remaining'] -= 1
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	771
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	772 self._collectionstack.append(
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	773 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	774 b'remaining': value,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	775 b'v': newvalue,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	776 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	777 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	778
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	779 self._state = self._STATE_WANT_ARRAY_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	780
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	781 # A new map is used as the map value.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	782 elif special == SPECIAL_START_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	783 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	784 newvalue = {}
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	785
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	786 lastc[b'v'][self._currentmapkey] = newvalue
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	787 lastc[b'remaining'] -= 1
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	788
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	789 self._collectionstack.append(
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	790 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	791 b'remaining': value,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	792 b'v': newvalue,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	793 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	794 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	795
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	796 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	797
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	798 # A new set is used as the map value.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	799 elif special == SPECIAL_START_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	800 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	801 newvalue = set()
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	802
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	803 lastc[b'v'][self._currentmapkey] = newvalue
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	804 lastc[b'remaining'] -= 1
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	805
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	806 self._collectionstack.append(
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	807 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	808 b'remaining': value,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	809 b'v': newvalue,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 43506 diff changeset	810 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	811 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	812
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	813 self._state = self._STATE_WANT_SET_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	814
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	815 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	816 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	817 b'indefinite length bytestrings not '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	818 b'allowed as map values'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	819 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	820
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	821 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	822 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	823 b'unhandled special item when '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	824 b'expecting map value: %d' % special
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	825 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	826
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	827 self._currentmapkey = None
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	828
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	829 # This value is added to the current set.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	830 elif self._state == self._STATE_WANT_SET_VALUE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	831 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	832 lastc = self._collectionstack[-1]
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	833 lastc[b'v'].add(value)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	834 lastc[b'remaining'] -= 1
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	835
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	836 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	837 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	838 b'indefinite length bytestrings not '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	839 b'allowed as set values'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	840 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	841
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	842 elif special in (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	843 SPECIAL_START_ARRAY,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	844 SPECIAL_START_MAP,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	845 SPECIAL_START_SET,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	846 ):
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	847 raise CBORDecodeError(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43106 diff changeset	848 b'collections not allowed as set values'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	849 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	850
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	851 # We don't allow non-trivial types to exist as set values.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	852 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	853 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	854 b'unhandled special item when '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	855 b'expecting set value: %d' % special
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	856 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	857
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	858 # This value represents the first chunk in an indefinite length
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	859 # bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	860 elif self._state == self._STATE_WANT_BYTESTRING_CHUNK_FIRST:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	861 # We received a full chunk.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	862 if special == SPECIAL_NONE:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	863 self._decodedvalues.append(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	864 bytestringchunk(value, first=True)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	865 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	866
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	867 self._state = self._STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	868
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	869 # The end of stream marker. This means it is an empty
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	870 # indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	871 elif special == SPECIAL_INDEFINITE_BREAK:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	872 # We /could/ convert this to a b''. But we want to preserve
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	873 # the nature of the underlying data so consumers expecting
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	874 # an indefinite length bytestring get one.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	875 self._decodedvalues.append(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	876 bytestringchunk(b'', first=True, last=True)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	877 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	878
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	879 # Since indefinite length bytestrings can't be used in
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	880 # collections, we must be at the root level.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	881 assert not self._collectionstack
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	882 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	883
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	884 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	885 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	886 b'unexpected special value when '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	887 b'expecting bytestring chunk: %d' % special
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	888 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	889
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	890 # This value represents the non-initial chunk in an indefinite
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	891 # length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	892 elif self._state == self._STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	893 # We received a full chunk.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	894 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	895 self._decodedvalues.append(bytestringchunk(value))
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	896
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	897 # The end of stream marker.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	898 elif special == SPECIAL_INDEFINITE_BREAK:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	899 self._decodedvalues.append(bytestringchunk(b'', last=True))
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	900
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	901 # Since indefinite length bytestrings can't be used in
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	902 # collections, we must be at the root level.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	903 assert not self._collectionstack
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	904 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	905
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	906 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	907 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	908 b'unexpected special value when '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	909 b'expecting bytestring chunk: %d' % special
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	910 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	911
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	912 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	913 raise CBORDecodeError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	914 b'unhandled decoder state: %d' % self._state
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	915 )
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	916
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	917 # We could have just added the final value in a collection. End
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	918 # all complete collections at the top of the stack.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	919 while True:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	920 # Bail if we're not waiting on a new collection item.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	921 if self._state not in (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	922 self._STATE_WANT_ARRAY_VALUE,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	923 self._STATE_WANT_MAP_KEY,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	924 self._STATE_WANT_SET_VALUE,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	925 ):
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	926 break
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	927
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	928 # Or we are expecting more items for this collection.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	929 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	930
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	931 if lastc[b'remaining']:
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	932 break
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	933
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	934 # The collection at the top of the stack is complete.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	935
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	936 # Discard it, as it isn't needed for future items.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	937 self._collectionstack.pop()
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	938
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	939 # If this is a nested collection, we don't emit it, since it
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	940 # will be emitted by its parent collection. But we do need to
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	941 # update state to reflect what the new top-most collection
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	942 # on the stack is.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	943 if self._collectionstack:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	944 self._state = {
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	945 list: self._STATE_WANT_ARRAY_VALUE,
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	946 dict: self._STATE_WANT_MAP_KEY,
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	947 set: self._STATE_WANT_SET_VALUE,
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	948 }[type(self._collectionstack[-1][b'v'])]
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	949
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	950 # If this is the root collection, emit it.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	951 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	952 self._decodedvalues.append(lastc[b'v'])
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	953 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	954
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	955 return (
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	956 bool(self._decodedvalues),
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	957 offset - initialoffset,
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	958 0,
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	959 )
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	960
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	961 def getavailable(self):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	962 """Returns an iterator over fully decoded values.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	963
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	964 Once values are retrieved, they won't be available on the next call.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	965 """
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	966
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	967 l = list(self._decodedvalues)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	968 self._decodedvalues = []
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	969 return l
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	970
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	971
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48942 diff changeset	972 class bufferingdecoder:
39413 babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	973 """A CBOR decoder that buffers undecoded input.
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	974
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	975 This is a glorified wrapper around ``sansiodecoder`` that adds a buffering
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	976 layer. All input that isn't consumed by ``sansiodecoder`` will be buffered
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	977 and concatenated with any new input that arrives later.
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	978
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	979 TODO consider adding limits as to the maximum amount of data that can
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	980 be buffered.
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	981 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	982
39413 babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	983 def __init__(self):
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	984 self._decoder = sansiodecoder()
40030 62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	985 self._chunks = []
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	986 self._wanted = 0
39413 babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	987
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	988 def decode(self, b):
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	989 """Attempt to decode bytes to CBOR values.
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	990
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	991 Returns a tuple with the following fields:
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	992
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	993 * Bool indicating whether new values are available for retrieval.
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	994 * Integer number of bytes decoded from the new input.
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	995 * Integer number of bytes wanted to decode the next value.
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	996 """
40124 b638219a23c3 cborutil: cast bytearray to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 40030 diff changeset	997 # We /might/ be able to support passing a bytearray all the
b638219a23c3 cborutil: cast bytearray to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 40030 diff changeset	998 # way through. For now, let's cheat.
b638219a23c3 cborutil: cast bytearray to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 40030 diff changeset	999 if isinstance(b, bytearray):
b638219a23c3 cborutil: cast bytearray to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 40030 diff changeset	1000 b = bytes(b)
b638219a23c3 cborutil: cast bytearray to bytes Gregory Szorc <gregory.szorc@gmail.com> parents: 40030 diff changeset	1001
40030 62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1002 # Our strategy for buffering is to aggregate the incoming chunks in a
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1003 # list until we've received enough data to decode the next item.
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1004 # This is slightly more complicated than using an ``io.BytesIO``
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1005 # or continuously concatenating incoming data. However, because it
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1006 # isn't constantly reallocating backing memory for a growing buffer,
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1007 # it prevents excessive memory thrashing and is significantly faster,
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1008 # especially in cases where the percentage of input chunks that don't
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1009 # decode into a full item is high.
39413 babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1010
40030 62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1011 if self._chunks:
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1012 # A previous call said we needed N bytes to decode the next item.
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1013 # But this call doesn't provide enough data. We buffer the incoming
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1014 # chunk without attempting to decode.
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1015 if len(b) < self._wanted:
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1016 self._chunks.append(b)
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1017 self._wanted -= len(b)
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1018 return False, 0, self._wanted
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1019
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1020 # Else we may have enough data to decode the next item. Aggregate
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1021 # old data with new and reset the buffer.
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1022 newlen = len(b)
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1023 self._chunks.append(b)
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1024 b = b''.join(self._chunks)
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1025 self._chunks = []
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1026 oldlen = len(b) - newlen
62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1027
39413 babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1028 else:
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1029 oldlen = 0
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1030
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1031 available, readcount, wanted = self._decoder.decode(b)
40030 62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1032 self._wanted = wanted
39413 babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1033
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1034 if readcount < len(b):
40030 62160d3077cd cborutil: change buffering strategy Gregory Szorc <gregory.szorc@gmail.com> parents: 39456 diff changeset	1035 self._chunks.append(b[readcount:])
39413 babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1036
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1037 return available, readcount - oldlen, wanted
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1038
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1039 def getavailable(self):
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1040 return self._decoder.getavailable()
babad5ebaf0a cborutil: add a buffering decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 39412 diff changeset	1041
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42480 diff changeset	1042
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1043 def decodeall(b):
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1044 """Decode all CBOR items present in an iterable of bytes.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1045
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1046 In addition to regular decode errors, raises CBORDecodeError if the
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1047 entirety of the passed buffer does not fully decode to complete CBOR
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1048 values. This includes failure to decode any value, incomplete collection
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1049 types, incomplete indefinite length items, and extra data at the end of
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1050 the buffer.
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1051 """
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1052 if not b:
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1053 return []
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1054
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1055 decoder = sansiodecoder()
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1056
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1057 havevalues, readcount, wantbytes = decoder.decode(b)
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1058
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1059 if readcount != len(b):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1060 raise CBORDecodeError(b'input data not fully consumed')
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1061
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1062 if decoder.inprogress:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1063 raise CBORDecodeError(b'input data not complete')
39411 aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1064
aeb551a3bb8a cborutil: implement sans I/O decoder Gregory Szorc <gregory.szorc@gmail.com> parents: 37898 diff changeset	1065 return decoder.getavailable()

Mercurial > public > mercurial-scm > hg

annotate mercurial/utils/cborutil.py @ 52644:e627cc25b6f3