Mercurial > public > mercurial-scm > hg

annotate mercurial/utils/cborutil.py @ 42648:d80edcb0b30c stable

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
automation: make Windows base image name configurable Since automation broke in the middle of the 5.0 release cycle, there's a good chance it will break again in the future. While a robust solution might be to search for all available images and choose the newest one, it does seem useful to be able to explicitly choose the name of the image to find and use so users can opt in to using a different image. This commit implements that functionality. Differential Revision: https://phab.mercurial-scm.org/D6673
author Gregory Szorc <gregory.szorc@gmail.com>
date Mon, 22 Jul 2019 19:06:20 -0700
parents b6387a65851d
children 2372284d9457
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # cborutil.py - CBOR extensions
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
8 from __future__ import absolute_import
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
10 import struct
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
11 import sys
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
12
39456
8d858fbf2759 cbor: teach the encoder to handle python `long` type for Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39413
diff changeset
13 from .. import pycompat
8d858fbf2759 cbor: teach the encoder to handle python `long` type for Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39413
diff changeset
14
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
15 # Very short very of RFC 7049...
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
16 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
17 # Each item begins with a byte. The 3 high bits of that byte denote the
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
18 # "major type." The lower 5 bits denote the "subtype." Each major type
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
19 # has its own encoding mechanism.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
20 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
21 # Most types have lengths. However, bytestring, string, array, and map
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
22 # can be indefinite length. These are denotes by a subtype with value 31.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
23 # Sub-components of those types then come afterwards and are terminated
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
24 # by a "break" byte.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
25
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
26 MAJOR_TYPE_UINT = 0
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
27 MAJOR_TYPE_NEGINT = 1
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
28 MAJOR_TYPE_BYTESTRING = 2
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
29 MAJOR_TYPE_STRING = 3
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
30 MAJOR_TYPE_ARRAY = 4
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
31 MAJOR_TYPE_MAP = 5
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
32 MAJOR_TYPE_SEMANTIC = 6
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
33 MAJOR_TYPE_SPECIAL = 7
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
34
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
35 SUBTYPE_MASK = 0b00011111
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
36
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
37 SUBTYPE_FALSE = 20
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
38 SUBTYPE_TRUE = 21
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
39 SUBTYPE_NULL = 22
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
40 SUBTYPE_HALF_FLOAT = 25
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
41 SUBTYPE_SINGLE_FLOAT = 26
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
42 SUBTYPE_DOUBLE_FLOAT = 27
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
43 SUBTYPE_INDEFINITE = 31
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
44
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
45 SEMANTIC_TAG_FINITE_SET = 258
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
46
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
47 # Indefinite types begin with their major type ORd with information value 31.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
48 BEGIN_INDEFINITE_BYTESTRING = struct.pack(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
49 r'>B', MAJOR_TYPE_BYTESTRING << 5 | SUBTYPE_INDEFINITE)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
50 BEGIN_INDEFINITE_ARRAY = struct.pack(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
51 r'>B', MAJOR_TYPE_ARRAY << 5 | SUBTYPE_INDEFINITE)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
52 BEGIN_INDEFINITE_MAP = struct.pack(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
53 r'>B', MAJOR_TYPE_MAP << 5 | SUBTYPE_INDEFINITE)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
54
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
55 ENCODED_LENGTH_1 = struct.Struct(r'>B')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
56 ENCODED_LENGTH_2 = struct.Struct(r'>BB')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
57 ENCODED_LENGTH_3 = struct.Struct(r'>BH')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
58 ENCODED_LENGTH_4 = struct.Struct(r'>BL')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
59 ENCODED_LENGTH_5 = struct.Struct(r'>BQ')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
60
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
61 # The break ends an indefinite length item.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
62 BREAK = b'\xff'
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
63 BREAK_INT = 255
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
64
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
65 def encodelength(majortype, length):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
66 """Obtain a value encoding the major type and its length."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
67 if length < 24:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
68 return ENCODED_LENGTH_1.pack(majortype << 5 | length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
69 elif length < 256:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
70 return ENCODED_LENGTH_2.pack(majortype << 5 | 24, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
71 elif length < 65536:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
72 return ENCODED_LENGTH_3.pack(majortype << 5 | 25, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
73 elif length < 4294967296:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
74 return ENCODED_LENGTH_4.pack(majortype << 5 | 26, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
75 else:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
76 return ENCODED_LENGTH_5.pack(majortype << 5 | 27, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
77
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
78 def streamencodebytestring(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
79 yield encodelength(MAJOR_TYPE_BYTESTRING, len(v))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
80 yield v
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
81
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
82 def streamencodebytestringfromiter(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
83 """Convert an iterator of chunks to an indefinite bytestring.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
84
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
85 Given an input that is iterable and each element in the iterator is
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
86 representable as bytes, emit an indefinite length bytestring.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
87 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
88 yield BEGIN_INDEFINITE_BYTESTRING
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
89
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
90 for chunk in it:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
91 yield encodelength(MAJOR_TYPE_BYTESTRING, len(chunk))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
92 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
93
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
94 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
95
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
96 def streamencodeindefinitebytestring(source, chunksize=65536):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
97 """Given a large source buffer, emit as an indefinite length bytestring.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
98
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
99 This is a generator of chunks constituting the encoded CBOR data.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
100 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
101 yield BEGIN_INDEFINITE_BYTESTRING
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
102
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
103 i = 0
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
104 l = len(source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
105
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
106 while True:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
107 chunk = source[i:i + chunksize]
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
108 i += len(chunk)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
109
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
110 yield encodelength(MAJOR_TYPE_BYTESTRING, len(chunk))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
111 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
112
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
113 if i >= l:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
114 break
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
115
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
116 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
117
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
118 def streamencodeint(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
119 if v >= 18446744073709551616 or v < -18446744073709551616:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
120 raise ValueError('big integers not supported')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
121
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
122 if v >= 0:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
123 yield encodelength(MAJOR_TYPE_UINT, v)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
124 else:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
125 yield encodelength(MAJOR_TYPE_NEGINT, abs(v) - 1)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
126
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
127 def streamencodearray(l):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
128 """Encode a known size iterable to an array."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
129
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
130 yield encodelength(MAJOR_TYPE_ARRAY, len(l))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
131
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
132 for i in l:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
133 for chunk in streamencode(i):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
134 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
135
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
136 def streamencodearrayfromiter(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
137 """Encode an iterator of items to an indefinite length array."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
138
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
139 yield BEGIN_INDEFINITE_ARRAY
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
140
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
141 for i in it:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
142 for chunk in streamencode(i):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
143 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
144
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
145 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
146
37898
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
147 def _mixedtypesortkey(v):
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
148 return type(v).__name__, v
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
149
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
150 def streamencodeset(s):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
151 # https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml defines
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
152 # semantic tag 258 for finite sets.
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
153 yield encodelength(MAJOR_TYPE_SEMANTIC, SEMANTIC_TAG_FINITE_SET)
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
154
37898
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
155 for chunk in streamencodearray(sorted(s, key=_mixedtypesortkey)):
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
156 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
157
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
158 def streamencodemap(d):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
159 """Encode dictionary to a generator.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
160
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
161 Does not supporting indefinite length dictionaries.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
162 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
163 yield encodelength(MAJOR_TYPE_MAP, len(d))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
164
37898
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
165 for key, value in sorted(d.iteritems(),
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
166 key=lambda x: _mixedtypesortkey(x[0])):
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
167 for chunk in streamencode(key):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
168 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
169 for chunk in streamencode(value):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
170 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
171
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
172 def streamencodemapfromiter(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
173 """Given an iterable of (key, value), encode to an indefinite length map."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
174 yield BEGIN_INDEFINITE_MAP
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
175
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
176 for key, value in it:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
177 for chunk in streamencode(key):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
178 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
179 for chunk in streamencode(value):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
180 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
181
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
182 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
183
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
184 def streamencodebool(b):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
185 # major type 7, simple value 20 and 21.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
186 yield b'\xf5' if b else b'\xf4'
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
187
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
188 def streamencodenone(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
189 # major type 7, simple value 22.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
190 yield b'\xf6'
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
191
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
192 STREAM_ENCODERS = {
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
193 bytes: streamencodebytestring,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
194 int: streamencodeint,
39456
8d858fbf2759 cbor: teach the encoder to handle python `long` type for Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39413
diff changeset
195 pycompat.long: streamencodeint,
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
196 list: streamencodearray,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
197 tuple: streamencodearray,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
198 dict: streamencodemap,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
199 set: streamencodeset,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
200 bool: streamencodebool,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
201 type(None): streamencodenone,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
202 }
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
203
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
204 def streamencode(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
205 """Encode a value in a streaming manner.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
206
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
207 Given an input object, encode it to CBOR recursively.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
208
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
209 Returns a generator of CBOR encoded bytes. There is no guarantee
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
210 that each emitted chunk fully decodes to a value or sub-value.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
211
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
212 Encoding is deterministic - unordered collections are sorted.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
213 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
214 fn = STREAM_ENCODERS.get(v.__class__)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
215
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
216 if not fn:
42480
b6387a65851d cborutil: fix streamencode() to handle subtypes
Yuya Nishihara <yuya@tcha.org>
parents: 40124
diff changeset
217 # handle subtypes such as encoding.localstr and util.sortdict
b6387a65851d cborutil: fix streamencode() to handle subtypes
Yuya Nishihara <yuya@tcha.org>
parents: 40124
diff changeset
218 for ty in STREAM_ENCODERS:
b6387a65851d cborutil: fix streamencode() to handle subtypes
Yuya Nishihara <yuya@tcha.org>
parents: 40124
diff changeset
219 if not isinstance(v, ty):
b6387a65851d cborutil: fix streamencode() to handle subtypes
Yuya Nishihara <yuya@tcha.org>
parents: 40124
diff changeset
220 continue
b6387a65851d cborutil: fix streamencode() to handle subtypes
Yuya Nishihara <yuya@tcha.org>
parents: 40124
diff changeset
221 fn = STREAM_ENCODERS[ty]
b6387a65851d cborutil: fix streamencode() to handle subtypes
Yuya Nishihara <yuya@tcha.org>
parents: 40124
diff changeset
222 break
b6387a65851d cborutil: fix streamencode() to handle subtypes
Yuya Nishihara <yuya@tcha.org>
parents: 40124
diff changeset
223
b6387a65851d cborutil: fix streamencode() to handle subtypes
Yuya Nishihara <yuya@tcha.org>
parents: 40124
diff changeset
224 if not fn:
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
225 raise ValueError('do not know how to encode %s' % type(v))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
226
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
227 return fn(v)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
228
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
229 class CBORDecodeError(Exception):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
230 """Represents an error decoding CBOR."""
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
231
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
232 if sys.version_info.major >= 3:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
233 def _elementtointeger(b, i):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
234 return b[i]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
235 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
236 def _elementtointeger(b, i):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
237 return ord(b[i])
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
238
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
239 STRUCT_BIG_UBYTE = struct.Struct(r'>B')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
240 STRUCT_BIG_USHORT = struct.Struct('>H')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
241 STRUCT_BIG_ULONG = struct.Struct('>L')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
242 STRUCT_BIG_ULONGLONG = struct.Struct('>Q')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
243
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
244 SPECIAL_NONE = 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
245 SPECIAL_START_INDEFINITE_BYTESTRING = 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
246 SPECIAL_START_ARRAY = 2
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
247 SPECIAL_START_MAP = 3
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
248 SPECIAL_START_SET = 4
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
249 SPECIAL_INDEFINITE_BREAK = 5
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
250
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
251 def decodeitem(b, offset=0):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
252 """Decode a new CBOR value from a buffer at offset.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
253
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
254 This function attempts to decode up to one complete CBOR value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
255 from ``b`` starting at offset ``offset``.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
256
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
257 The beginning of a collection (such as an array, map, set, or
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
258 indefinite length bytestring) counts as a single value. For these
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
259 special cases, a state flag will indicate that a special value was seen.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
260
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
261 When called, the function either returns a decoded value or gives
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
262 a hint as to how many more bytes are needed to do so. By calling
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
263 the function repeatedly given a stream of bytes, the caller can
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
264 build up the original values.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
265
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
266 Returns a tuple with the following elements:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
267
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
268 * Bool indicating whether a complete value was decoded.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
269 * A decoded value if first value is True otherwise None
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
270 * Integer number of bytes. If positive, the number of bytes
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
271 read. If negative, the number of bytes we need to read to
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
272 decode this value or the next chunk in this value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
273 * One of the ``SPECIAL_*`` constants indicating special treatment
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
274 for this value. ``SPECIAL_NONE`` means this is a fully decoded
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
275 simple value (such as an integer or bool).
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
276 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
277
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
278 initial = _elementtointeger(b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
279 offset += 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
280
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
281 majortype = initial >> 5
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
282 subtype = initial & SUBTYPE_MASK
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
283
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
284 if majortype == MAJOR_TYPE_UINT:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
285 complete, value, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
286
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
287 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
288 return True, value, readcount + 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
289 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
290 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
291
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
292 elif majortype == MAJOR_TYPE_NEGINT:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
293 # Negative integers are the same as UINT except inverted minus 1.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
294 complete, value, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
295
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
296 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
297 return True, -value - 1, readcount + 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
298 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
299 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
300
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
301 elif majortype == MAJOR_TYPE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
302 # Beginning of bytestrings are treated as uints in order to
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
303 # decode their length, which may be indefinite.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
304 complete, size, readcount = decodeuint(subtype, b, offset,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
305 allowindefinite=True)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
306
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
307 # We don't know the size of the bytestring. It must be a definitive
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
308 # length since the indefinite subtype would be encoded in the initial
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
309 # byte.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
310 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
311 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
312
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
313 # We know the length of the bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
314 if size is not None:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
315 # And the data is available in the buffer.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
316 if offset + readcount + size <= len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
317 value = b[offset + readcount:offset + readcount + size]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
318 return True, value, readcount + size + 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
319
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
320 # And we need more data in order to return the bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
321 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
322 wanted = len(b) - offset - readcount - size
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
323 return False, None, wanted, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
324
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
325 # It is an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
326 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
327 return True, None, 1, SPECIAL_START_INDEFINITE_BYTESTRING
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
328
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
329 elif majortype == MAJOR_TYPE_STRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
330 raise CBORDecodeError('string major type not supported')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
331
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
332 elif majortype == MAJOR_TYPE_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
333 # Beginning of arrays are treated as uints in order to decode their
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
334 # length. We don't allow indefinite length arrays.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
335 complete, size, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
336
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
337 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
338 return True, size, readcount + 1, SPECIAL_START_ARRAY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
339 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
340 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
341
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
342 elif majortype == MAJOR_TYPE_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
343 # Beginning of maps are treated as uints in order to decode their
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
344 # number of elements. We don't allow indefinite length arrays.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
345 complete, size, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
346
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
347 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
348 return True, size, readcount + 1, SPECIAL_START_MAP
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
349 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
350 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
351
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
352 elif majortype == MAJOR_TYPE_SEMANTIC:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
353 # Semantic tag value is read the same as a uint.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
354 complete, tagvalue, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
355
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
356 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
357 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
358
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
359 # This behavior here is a little wonky. The main type being "decorated"
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
360 # by this semantic tag follows. A more robust parser would probably emit
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
361 # a special flag indicating this as a semantic tag and let the caller
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
362 # deal with the types that follow. But since we don't support many
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
363 # semantic tags, it is easier to deal with the special cases here and
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
364 # hide complexity from the caller. If we add support for more semantic
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
365 # tags, we should probably move semantic tag handling into the caller.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
366 if tagvalue == SEMANTIC_TAG_FINITE_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
367 if offset + readcount >= len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
368 return False, None, -1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
369
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
370 complete, size, readcount2, special = decodeitem(b,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
371 offset + readcount)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
372
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
373 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
374 return False, None, readcount2, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
375
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
376 if special != SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
377 raise CBORDecodeError('expected array after finite set '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
378 'semantic tag')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
379
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
380 return True, size, readcount + readcount2 + 1, SPECIAL_START_SET
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
381
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
382 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
383 raise CBORDecodeError('semantic tag %d not allowed' % tagvalue)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
384
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
385 elif majortype == MAJOR_TYPE_SPECIAL:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
386 # Only specific values for the information field are allowed.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
387 if subtype == SUBTYPE_FALSE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
388 return True, False, 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
389 elif subtype == SUBTYPE_TRUE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
390 return True, True, 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
391 elif subtype == SUBTYPE_NULL:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
392 return True, None, 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
393 elif subtype == SUBTYPE_INDEFINITE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
394 return True, None, 1, SPECIAL_INDEFINITE_BREAK
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
395 # If value is 24, subtype is in next byte.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
396 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
397 raise CBORDecodeError('special type %d not allowed' % subtype)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
398 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
399 assert False
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
400
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
401 def decodeuint(subtype, b, offset=0, allowindefinite=False):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
402 """Decode an unsigned integer.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
403
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
404 ``subtype`` is the lower 5 bits from the initial byte CBOR item
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
405 "header." ``b`` is a buffer containing bytes. ``offset`` points to
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
406 the index of the first byte after the byte that ``subtype`` was
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
407 derived from.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
408
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
409 ``allowindefinite`` allows the special indefinite length value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
410 indicator.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
412 Returns a 3-tuple of (successful, value, count).
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
413
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
414 The first element is a bool indicating if decoding completed. The 2nd
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
415 is the decoded integer value or None if not fully decoded or the subtype
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
416 is 31 and ``allowindefinite`` is True. The 3rd value is the count of bytes.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
417 If positive, it is the number of additional bytes decoded. If negative,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
418 it is the number of additional bytes needed to decode this value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
419 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
420
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
421 # Small values are inline.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
422 if subtype < 24:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
423 return True, subtype, 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
424 # Indefinite length specifier.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
425 elif subtype == 31:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
426 if allowindefinite:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
427 return True, None, 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
428 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
429 raise CBORDecodeError('indefinite length uint not allowed here')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
430 elif subtype >= 28:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
431 raise CBORDecodeError('unsupported subtype on integer type: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
432 subtype)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
433
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
434 if subtype == 24:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
435 s = STRUCT_BIG_UBYTE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
436 elif subtype == 25:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
437 s = STRUCT_BIG_USHORT
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
438 elif subtype == 26:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
439 s = STRUCT_BIG_ULONG
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
440 elif subtype == 27:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
441 s = STRUCT_BIG_ULONGLONG
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
442 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
443 raise CBORDecodeError('bounds condition checking violation')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
444
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
445 if len(b) - offset >= s.size:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
446 return True, s.unpack_from(b, offset)[0], s.size
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
447 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
448 return False, None, len(b) - offset - s.size
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
449
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
450 class bytestringchunk(bytes):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
451 """Represents a chunk/segment in an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
452
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
453 This behaves like a ``bytes`` but in addition has the ``isfirst``
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
454 and ``islast`` attributes indicating whether this chunk is the first
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
455 or last in an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
456 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
457
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
458 def __new__(cls, v, first=False, last=False):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
459 self = bytes.__new__(cls, v)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
460 self.isfirst = first
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
461 self.islast = last
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
462
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
463 return self
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
464
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
465 class sansiodecoder(object):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
466 """A CBOR decoder that doesn't perform its own I/O.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
467
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
468 To use, construct an instance and feed it segments containing
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
469 CBOR-encoded bytes via ``decode()``. The return value from ``decode()``
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
470 indicates whether a fully-decoded value is available, how many bytes
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
471 were consumed, and offers a hint as to how many bytes should be fed
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
472 in next time to decode the next value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
473
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
474 The decoder assumes it will decode N discrete CBOR values, not just
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
475 a single value. i.e. if the bytestream contains uints packed one after
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
476 the other, the decoder will decode them all, rather than just the initial
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
477 one.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
478
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
479 When ``decode()`` indicates a value is available, call ``getavailable()``
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
480 to return all fully decoded values.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
481
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
482 ``decode()`` can partially decode input. It is up to the caller to keep
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
483 track of what data was consumed and to pass unconsumed data in on the
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
484 next invocation.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
485
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
486 The decoder decodes atomically at the *item* level. See ``decodeitem()``.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
487 If an *item* cannot be fully decoded, the decoder won't record it as
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
488 partially consumed. Instead, the caller will be instructed to pass in
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
489 the initial bytes of this item on the next invocation. This does result
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
490 in some redundant parsing. But the overhead should be minimal.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
491
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
492 This decoder only supports a subset of CBOR as required by Mercurial.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
493 It lacks support for:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
494
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
495 * Indefinite length arrays
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
496 * Indefinite length maps
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
497 * Use of indefinite length bytestrings as keys or values within
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
498 arrays, maps, or sets.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
499 * Nested arrays, maps, or sets within sets
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
500 * Any semantic tag that isn't a mathematical finite set
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
501 * Floating point numbers
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
502 * Undefined special value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
503
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
504 CBOR types are decoded to Python types as follows:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
505
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
506 uint -> int
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
507 negint -> int
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
508 bytestring -> bytes
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
509 map -> dict
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
510 array -> list
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
511 True -> bool
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
512 False -> bool
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
513 null -> None
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
514 indefinite length bytestring chunk -> [bytestringchunk]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
515
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
516 The only non-obvious mapping here is an indefinite length bytestring
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
517 to the ``bytestringchunk`` type. This is to facilitate streaming
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
518 indefinite length bytestrings out of the decoder and to differentiate
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
519 a regular bytestring from an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
520 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
521
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
522 _STATE_NONE = 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
523 _STATE_WANT_MAP_KEY = 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
524 _STATE_WANT_MAP_VALUE = 2
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
525 _STATE_WANT_ARRAY_VALUE = 3
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
526 _STATE_WANT_SET_VALUE = 4
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
527 _STATE_WANT_BYTESTRING_CHUNK_FIRST = 5
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
528 _STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT = 6
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
529
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
530 def __init__(self):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
531 # TODO add support for limiting size of bytestrings
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
532 # TODO add support for limiting number of keys / values in collections
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
533 # TODO add support for limiting size of buffered partial values
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
534
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
535 self.decodedbytecount = 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
536
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
537 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
538
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
539 # Stack of active nested collections. Each entry is a dict describing
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
540 # the collection.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
541 self._collectionstack = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
542
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
543 # Fully decoded key to use for the current map.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
544 self._currentmapkey = None
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
545
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
546 # Fully decoded values available for retrieval.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
547 self._decodedvalues = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
548
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
549 @property
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
550 def inprogress(self):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
551 """Whether the decoder has partially decoded a value."""
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
552 return self._state != self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
553
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
554 def decode(self, b, offset=0):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
555 """Attempt to decode bytes from an input buffer.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
556
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
557 ``b`` is a collection of bytes and ``offset`` is the byte
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
558 offset within that buffer from which to begin reading data.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
559
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
560 ``b`` must support ``len()`` and accessing bytes slices via
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
561 ``__slice__``. Typically ``bytes`` instances are used.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
562
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
563 Returns a tuple with the following fields:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
564
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
565 * Bool indicating whether values are available for retrieval.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
566 * Integer indicating the number of bytes that were fully consumed,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
567 starting from ``offset``.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
568 * Integer indicating the number of bytes that are desired for the
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
569 next call in order to decode an item.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
570 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
571 if not b:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
572 return bool(self._decodedvalues), 0, 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
573
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
574 initialoffset = offset
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
575
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
576 # We could easily split the body of this loop into a function. But
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
577 # Python performance is sensitive to function calls and collections
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
578 # are composed of many items. So leaving as a while loop could help
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
579 # with performance. One thing that may not help is the use of
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
580 # if..elif versus a lookup/dispatch table. There may be value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
581 # in switching that.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
582 while offset < len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
583 # Attempt to decode an item. This could be a whole value or a
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
584 # special value indicating an event, such as start or end of a
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
585 # collection or indefinite length type.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
586 complete, value, readcount, special = decodeitem(b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
587
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
588 if readcount > 0:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
589 self.decodedbytecount += readcount
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
590
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
591 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
592 assert readcount < 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
593 return (
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
594 bool(self._decodedvalues),
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
595 offset - initialoffset,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
596 -readcount,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
597 )
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
598
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
599 offset += readcount
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
600
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
601 # No nested state. We either have a full value or beginning of a
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
602 # complex value to deal with.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
603 if self._state == self._STATE_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
604 # A normal value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
605 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
606 self._decodedvalues.append(value)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
607
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
608 elif special == SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
609 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
610 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
611 'v': [],
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
612 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
613 self._state = self._STATE_WANT_ARRAY_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
614
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
615 elif special == SPECIAL_START_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
616 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
617 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
618 'v': {},
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
619 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
620 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
621
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
622 elif special == SPECIAL_START_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
623 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
624 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
625 'v': set(),
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
626 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
627 self._state = self._STATE_WANT_SET_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
628
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
629 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
630 self._state = self._STATE_WANT_BYTESTRING_CHUNK_FIRST
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
631
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
632 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
633 raise CBORDecodeError('unhandled special state: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
634 special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
635
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
636 # This value becomes an element of the current array.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
637 elif self._state == self._STATE_WANT_ARRAY_VALUE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
638 # Simple values get appended.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
639 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
640 c = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
641 c['v'].append(value)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
642 c['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
643
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
644 # self._state doesn't need changed.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
645
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
646 # An array nested within an array.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
647 elif special == SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
648 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
649 newvalue = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
650
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
651 lastc['v'].append(newvalue)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
652 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
653
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
654 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
655 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
656 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
657 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
658
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
659 # self._state doesn't need changed.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
660
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
661 # A map nested within an array.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
662 elif special == SPECIAL_START_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
663 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
664 newvalue = {}
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
665
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
666 lastc['v'].append(newvalue)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
667 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
668
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
669 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
670 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
671 'v': newvalue
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
672 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
673
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
674 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
675
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
676 elif special == SPECIAL_START_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
677 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
678 newvalue = set()
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
679
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
680 lastc['v'].append(newvalue)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
681 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
682
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
683 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
684 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
685 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
686 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
687
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
688 self._state = self._STATE_WANT_SET_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
689
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
690 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
691 raise CBORDecodeError('indefinite length bytestrings '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
692 'not allowed as array values')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
693
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
694 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
695 raise CBORDecodeError('unhandled special item when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
696 'expecting array value: %d' % special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
697
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
698 # This value becomes the key of the current map instance.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
699 elif self._state == self._STATE_WANT_MAP_KEY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
700 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
701 self._currentmapkey = value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
702 self._state = self._STATE_WANT_MAP_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
703
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
704 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
705 raise CBORDecodeError('indefinite length bytestrings '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
706 'not allowed as map keys')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
707
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
708 elif special in (SPECIAL_START_ARRAY, SPECIAL_START_MAP,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
709 SPECIAL_START_SET):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
710 raise CBORDecodeError('collections not supported as map '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
711 'keys')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
712
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
713 # We do not allow special values to be used as map keys.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
714 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
715 raise CBORDecodeError('unhandled special item when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
716 'expecting map key: %d' % special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
717
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
718 # This value becomes the value of the current map key.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
719 elif self._state == self._STATE_WANT_MAP_VALUE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
720 # Simple values simply get inserted into the map.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
721 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
722 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
723 lastc['v'][self._currentmapkey] = value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
724 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
725
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
726 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
727
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
728 # A new array is used as the map value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
729 elif special == SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
730 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
731 newvalue = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
732
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
733 lastc['v'][self._currentmapkey] = newvalue
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
734 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
735
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
736 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
737 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
738 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
739 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
740
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
741 self._state = self._STATE_WANT_ARRAY_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
742
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
743 # A new map is used as the map value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
744 elif special == SPECIAL_START_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
745 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
746 newvalue = {}
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
747
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
748 lastc['v'][self._currentmapkey] = newvalue
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
749 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
750
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
751 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
752 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
753 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
754 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
755
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
756 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
757
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
758 # A new set is used as the map value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
759 elif special == SPECIAL_START_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
760 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
761 newvalue = set()
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
762
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
763 lastc['v'][self._currentmapkey] = newvalue
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
764 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
765
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
766 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
767 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
768 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
769 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
770
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
771 self._state = self._STATE_WANT_SET_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
772
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
773 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
774 raise CBORDecodeError('indefinite length bytestrings not '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
775 'allowed as map values')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
776
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
777 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
778 raise CBORDecodeError('unhandled special item when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
779 'expecting map value: %d' % special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
780
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
781 self._currentmapkey = None
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
782
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
783 # This value is added to the current set.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
784 elif self._state == self._STATE_WANT_SET_VALUE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
785 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
786 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
787 lastc['v'].add(value)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
788 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
789
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
790 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
791 raise CBORDecodeError('indefinite length bytestrings not '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
792 'allowed as set values')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
793
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
794 elif special in (SPECIAL_START_ARRAY,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
795 SPECIAL_START_MAP,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
796 SPECIAL_START_SET):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
797 raise CBORDecodeError('collections not allowed as set '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
798 'values')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
799
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
800 # We don't allow non-trivial types to exist as set values.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
801 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
802 raise CBORDecodeError('unhandled special item when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
803 'expecting set value: %d' % special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
804
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
805 # This value represents the first chunk in an indefinite length
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
806 # bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
807 elif self._state == self._STATE_WANT_BYTESTRING_CHUNK_FIRST:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
808 # We received a full chunk.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
809 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
810 self._decodedvalues.append(bytestringchunk(value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
811 first=True))
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
812
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
813 self._state = self._STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
814
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
815 # The end of stream marker. This means it is an empty
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
816 # indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
817 elif special == SPECIAL_INDEFINITE_BREAK:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
818 # We /could/ convert this to a b''. But we want to preserve
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
819 # the nature of the underlying data so consumers expecting
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
820 # an indefinite length bytestring get one.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
821 self._decodedvalues.append(bytestringchunk(b'',
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
822 first=True,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
823 last=True))
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
824
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
825 # Since indefinite length bytestrings can't be used in
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
826 # collections, we must be at the root level.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
827 assert not self._collectionstack
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
828 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
829
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
830 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
831 raise CBORDecodeError('unexpected special value when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
832 'expecting bytestring chunk: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
833 special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
834
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
835 # This value represents the non-initial chunk in an indefinite
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
836 # length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
837 elif self._state == self._STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
838 # We received a full chunk.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
839 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
840 self._decodedvalues.append(bytestringchunk(value))
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
841
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
842 # The end of stream marker.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
843 elif special == SPECIAL_INDEFINITE_BREAK:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
844 self._decodedvalues.append(bytestringchunk(b'', last=True))
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
845
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
846 # Since indefinite length bytestrings can't be used in
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
847 # collections, we must be at the root level.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
848 assert not self._collectionstack
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
849 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
850
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
851 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
852 raise CBORDecodeError('unexpected special value when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
853 'expecting bytestring chunk: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
854 special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
855
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
856 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
857 raise CBORDecodeError('unhandled decoder state: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
858 self._state)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
859
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
860 # We could have just added the final value in a collection. End
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
861 # all complete collections at the top of the stack.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
862 while True:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
863 # Bail if we're not waiting on a new collection item.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
864 if self._state not in (self._STATE_WANT_ARRAY_VALUE,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
865 self._STATE_WANT_MAP_KEY,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
866 self._STATE_WANT_SET_VALUE):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
867 break
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
868
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
869 # Or we are expecting more items for this collection.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
870 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
871
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
872 if lastc['remaining']:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
873 break
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
874
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
875 # The collection at the top of the stack is complete.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
876
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
877 # Discard it, as it isn't needed for future items.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
878 self._collectionstack.pop()
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
879
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
880 # If this is a nested collection, we don't emit it, since it
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
881 # will be emitted by its parent collection. But we do need to
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
882 # update state to reflect what the new top-most collection
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
883 # on the stack is.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
884 if self._collectionstack:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
885 self._state = {
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
886 list: self._STATE_WANT_ARRAY_VALUE,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
887 dict: self._STATE_WANT_MAP_KEY,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
888 set: self._STATE_WANT_SET_VALUE,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
889 }[type(self._collectionstack[-1]['v'])]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
890
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
891 # If this is the root collection, emit it.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
892 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
893 self._decodedvalues.append(lastc['v'])
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
894 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
895
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
896 return (
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
897 bool(self._decodedvalues),
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
898 offset - initialoffset,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
899 0,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
900 )
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
901
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
902 def getavailable(self):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
903 """Returns an iterator over fully decoded values.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
904
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
905 Once values are retrieved, they won't be available on the next call.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
906 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
907
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
908 l = list(self._decodedvalues)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
909 self._decodedvalues = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
910 return l
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
911
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
912 class bufferingdecoder(object):
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
913 """A CBOR decoder that buffers undecoded input.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
914
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
915 This is a glorified wrapper around ``sansiodecoder`` that adds a buffering
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
916 layer. All input that isn't consumed by ``sansiodecoder`` will be buffered
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
917 and concatenated with any new input that arrives later.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
918
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
919 TODO consider adding limits as to the maximum amount of data that can
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
920 be buffered.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
921 """
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
922 def __init__(self):
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
923 self._decoder = sansiodecoder()
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
924 self._chunks = []
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
925 self._wanted = 0
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
926
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
927 def decode(self, b):
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
928 """Attempt to decode bytes to CBOR values.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
929
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
930 Returns a tuple with the following fields:
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
931
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
932 * Bool indicating whether new values are available for retrieval.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
933 * Integer number of bytes decoded from the new input.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
934 * Integer number of bytes wanted to decode the next value.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
935 """
40124
b638219a23c3 cborutil: cast bytearray to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40030
diff changeset
936 # We /might/ be able to support passing a bytearray all the
b638219a23c3 cborutil: cast bytearray to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40030
diff changeset
937 # way through. For now, let's cheat.
b638219a23c3 cborutil: cast bytearray to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40030
diff changeset
938 if isinstance(b, bytearray):
b638219a23c3 cborutil: cast bytearray to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40030
diff changeset
939 b = bytes(b)
b638219a23c3 cborutil: cast bytearray to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40030
diff changeset
940
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
941 # Our strategy for buffering is to aggregate the incoming chunks in a
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
942 # list until we've received enough data to decode the next item.
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
943 # This is slightly more complicated than using an ``io.BytesIO``
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
944 # or continuously concatenating incoming data. However, because it
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
945 # isn't constantly reallocating backing memory for a growing buffer,
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
946 # it prevents excessive memory thrashing and is significantly faster,
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
947 # especially in cases where the percentage of input chunks that don't
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
948 # decode into a full item is high.
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
949
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
950 if self._chunks:
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
951 # A previous call said we needed N bytes to decode the next item.
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
952 # But this call doesn't provide enough data. We buffer the incoming
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
953 # chunk without attempting to decode.
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
954 if len(b) < self._wanted:
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
955 self._chunks.append(b)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
956 self._wanted -= len(b)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
957 return False, 0, self._wanted
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
958
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
959 # Else we may have enough data to decode the next item. Aggregate
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
960 # old data with new and reset the buffer.
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
961 newlen = len(b)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
962 self._chunks.append(b)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
963 b = b''.join(self._chunks)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
964 self._chunks = []
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
965 oldlen = len(b) - newlen
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
966
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
967 else:
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
968 oldlen = 0
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
969
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
970 available, readcount, wanted = self._decoder.decode(b)
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
971 self._wanted = wanted
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
972
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
973 if readcount < len(b):
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
974 self._chunks.append(b[readcount:])
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
975
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
976 return available, readcount - oldlen, wanted
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
977
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
978 def getavailable(self):
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
979 return self._decoder.getavailable()
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
980
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
981 def decodeall(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
982 """Decode all CBOR items present in an iterable of bytes.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
983
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
984 In addition to regular decode errors, raises CBORDecodeError if the
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
985 entirety of the passed buffer does not fully decode to complete CBOR
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
986 values. This includes failure to decode any value, incomplete collection
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
987 types, incomplete indefinite length items, and extra data at the end of
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
988 the buffer.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
989 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
990 if not b:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
991 return []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
992
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
993 decoder = sansiodecoder()
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
994
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
995 havevalues, readcount, wantbytes = decoder.decode(b)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
996
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
997 if readcount != len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
998 raise CBORDecodeError('input data not fully consumed')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
999
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
1000 if decoder.inprogress:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
1001 raise CBORDecodeError('input data not complete')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
1002
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
1003 return decoder.getavailable()