annotate mercurial/utils/cborutil.py @ 40030:62160d3077cd

cborutil: change buffering strategy Profiling revealed that we were spending a lot of time on the line that was concatenating the old buffer with the incoming data when attempting to decode long byte strings, such as manifest revisions. Essentially, we were feeding N chunks of size len(X) << len(Y) into decode() and continuously allocating a new, larger buffer to hold the undecoded input. This created substantial memory churn and slowed down execution. Changing the code to aggregate pending chunks in a list until we have enough data to fully decode the next atom makes things much more efficient. I don't have exact data, but I recall the old code spending >1s on manifest fulltexts from the mozilla-unified repo. The new code doesn't significantly appear in profile output. Differential Revision: https://phab.mercurial-scm.org/D4854
author Gregory Szorc <gregory.szorc@gmail.com>
date Wed, 03 Oct 2018 09:43:01 -0700
parents 8d858fbf2759
children b638219a23c3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # cborutil.py - CBOR extensions
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
8 from __future__ import absolute_import
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
10 import struct
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
11 import sys
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
12
39456
8d858fbf2759 cbor: teach the encoder to handle python `long` type for Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39413
diff changeset
13 from .. import pycompat
8d858fbf2759 cbor: teach the encoder to handle python `long` type for Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39413
diff changeset
14
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
15 # Very short very of RFC 7049...
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
16 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
17 # Each item begins with a byte. The 3 high bits of that byte denote the
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
18 # "major type." The lower 5 bits denote the "subtype." Each major type
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
19 # has its own encoding mechanism.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
20 #
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
21 # Most types have lengths. However, bytestring, string, array, and map
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
22 # can be indefinite length. These are denotes by a subtype with value 31.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
23 # Sub-components of those types then come afterwards and are terminated
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
24 # by a "break" byte.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
25
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
26 MAJOR_TYPE_UINT = 0
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
27 MAJOR_TYPE_NEGINT = 1
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
28 MAJOR_TYPE_BYTESTRING = 2
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
29 MAJOR_TYPE_STRING = 3
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
30 MAJOR_TYPE_ARRAY = 4
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
31 MAJOR_TYPE_MAP = 5
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
32 MAJOR_TYPE_SEMANTIC = 6
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
33 MAJOR_TYPE_SPECIAL = 7
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
34
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
35 SUBTYPE_MASK = 0b00011111
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
36
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
37 SUBTYPE_FALSE = 20
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
38 SUBTYPE_TRUE = 21
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
39 SUBTYPE_NULL = 22
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
40 SUBTYPE_HALF_FLOAT = 25
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
41 SUBTYPE_SINGLE_FLOAT = 26
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
42 SUBTYPE_DOUBLE_FLOAT = 27
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
43 SUBTYPE_INDEFINITE = 31
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
44
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
45 SEMANTIC_TAG_FINITE_SET = 258
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
46
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
47 # Indefinite types begin with their major type ORd with information value 31.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
48 BEGIN_INDEFINITE_BYTESTRING = struct.pack(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
49 r'>B', MAJOR_TYPE_BYTESTRING << 5 | SUBTYPE_INDEFINITE)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
50 BEGIN_INDEFINITE_ARRAY = struct.pack(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
51 r'>B', MAJOR_TYPE_ARRAY << 5 | SUBTYPE_INDEFINITE)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
52 BEGIN_INDEFINITE_MAP = struct.pack(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
53 r'>B', MAJOR_TYPE_MAP << 5 | SUBTYPE_INDEFINITE)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
54
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
55 ENCODED_LENGTH_1 = struct.Struct(r'>B')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
56 ENCODED_LENGTH_2 = struct.Struct(r'>BB')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
57 ENCODED_LENGTH_3 = struct.Struct(r'>BH')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
58 ENCODED_LENGTH_4 = struct.Struct(r'>BL')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
59 ENCODED_LENGTH_5 = struct.Struct(r'>BQ')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
60
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
61 # The break ends an indefinite length item.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
62 BREAK = b'\xff'
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
63 BREAK_INT = 255
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
64
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
65 def encodelength(majortype, length):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
66 """Obtain a value encoding the major type and its length."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
67 if length < 24:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
68 return ENCODED_LENGTH_1.pack(majortype << 5 | length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
69 elif length < 256:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
70 return ENCODED_LENGTH_2.pack(majortype << 5 | 24, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
71 elif length < 65536:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
72 return ENCODED_LENGTH_3.pack(majortype << 5 | 25, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
73 elif length < 4294967296:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
74 return ENCODED_LENGTH_4.pack(majortype << 5 | 26, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
75 else:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
76 return ENCODED_LENGTH_5.pack(majortype << 5 | 27, length)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
77
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
78 def streamencodebytestring(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
79 yield encodelength(MAJOR_TYPE_BYTESTRING, len(v))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
80 yield v
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
81
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
82 def streamencodebytestringfromiter(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
83 """Convert an iterator of chunks to an indefinite bytestring.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
84
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
85 Given an input that is iterable and each element in the iterator is
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
86 representable as bytes, emit an indefinite length bytestring.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
87 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
88 yield BEGIN_INDEFINITE_BYTESTRING
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
89
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
90 for chunk in it:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
91 yield encodelength(MAJOR_TYPE_BYTESTRING, len(chunk))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
92 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
93
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
94 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
95
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
96 def streamencodeindefinitebytestring(source, chunksize=65536):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
97 """Given a large source buffer, emit as an indefinite length bytestring.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
98
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
99 This is a generator of chunks constituting the encoded CBOR data.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
100 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
101 yield BEGIN_INDEFINITE_BYTESTRING
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
102
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
103 i = 0
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
104 l = len(source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
105
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
106 while True:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
107 chunk = source[i:i + chunksize]
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
108 i += len(chunk)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
109
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
110 yield encodelength(MAJOR_TYPE_BYTESTRING, len(chunk))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
111 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
112
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
113 if i >= l:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
114 break
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
115
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
116 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
117
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
118 def streamencodeint(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
119 if v >= 18446744073709551616 or v < -18446744073709551616:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
120 raise ValueError('big integers not supported')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
121
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
122 if v >= 0:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
123 yield encodelength(MAJOR_TYPE_UINT, v)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
124 else:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
125 yield encodelength(MAJOR_TYPE_NEGINT, abs(v) - 1)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
126
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
127 def streamencodearray(l):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
128 """Encode a known size iterable to an array."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
129
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
130 yield encodelength(MAJOR_TYPE_ARRAY, len(l))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
131
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
132 for i in l:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
133 for chunk in streamencode(i):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
134 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
135
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
136 def streamencodearrayfromiter(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
137 """Encode an iterator of items to an indefinite length array."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
138
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
139 yield BEGIN_INDEFINITE_ARRAY
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
140
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
141 for i in it:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
142 for chunk in streamencode(i):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
143 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
144
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
145 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
146
37898
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
147 def _mixedtypesortkey(v):
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
148 return type(v).__name__, v
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
149
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
150 def streamencodeset(s):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
151 # https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml defines
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
152 # semantic tag 258 for finite sets.
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
153 yield encodelength(MAJOR_TYPE_SEMANTIC, SEMANTIC_TAG_FINITE_SET)
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
154
37898
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
155 for chunk in streamencodearray(sorted(s, key=_mixedtypesortkey)):
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
156 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
157
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
158 def streamencodemap(d):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
159 """Encode dictionary to a generator.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
160
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
161 Does not supporting indefinite length dictionaries.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
162 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
163 yield encodelength(MAJOR_TYPE_MAP, len(d))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
164
37898
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
165 for key, value in sorted(d.iteritems(),
2ae6a3134362 cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents: 37711
diff changeset
166 key=lambda x: _mixedtypesortkey(x[0])):
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
167 for chunk in streamencode(key):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
168 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
169 for chunk in streamencode(value):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
170 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
171
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
172 def streamencodemapfromiter(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
173 """Given an iterable of (key, value), encode to an indefinite length map."""
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
174 yield BEGIN_INDEFINITE_MAP
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
175
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
176 for key, value in it:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
177 for chunk in streamencode(key):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
178 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
179 for chunk in streamencode(value):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
180 yield chunk
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
181
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
182 yield BREAK
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
183
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
184 def streamencodebool(b):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
185 # major type 7, simple value 20 and 21.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
186 yield b'\xf5' if b else b'\xf4'
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
187
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
188 def streamencodenone(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
189 # major type 7, simple value 22.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
190 yield b'\xf6'
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
191
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
192 STREAM_ENCODERS = {
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
193 bytes: streamencodebytestring,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
194 int: streamencodeint,
39456
8d858fbf2759 cbor: teach the encoder to handle python `long` type for Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39413
diff changeset
195 pycompat.long: streamencodeint,
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
196 list: streamencodearray,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
197 tuple: streamencodearray,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
198 dict: streamencodemap,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
199 set: streamencodeset,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
200 bool: streamencodebool,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
201 type(None): streamencodenone,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
202 }
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
203
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
204 def streamencode(v):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
205 """Encode a value in a streaming manner.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
206
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
207 Given an input object, encode it to CBOR recursively.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
208
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
209 Returns a generator of CBOR encoded bytes. There is no guarantee
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
210 that each emitted chunk fully decodes to a value or sub-value.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
211
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
212 Encoding is deterministic - unordered collections are sorted.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
213 """
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
214 fn = STREAM_ENCODERS.get(v.__class__)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
215
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
216 if not fn:
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
217 raise ValueError('do not know how to encode %s' % type(v))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
218
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
219 return fn(v)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
220
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
221 class CBORDecodeError(Exception):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
222 """Represents an error decoding CBOR."""
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
223
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
224 if sys.version_info.major >= 3:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
225 def _elementtointeger(b, i):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
226 return b[i]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
227 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
228 def _elementtointeger(b, i):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
229 return ord(b[i])
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
230
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
231 STRUCT_BIG_UBYTE = struct.Struct(r'>B')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
232 STRUCT_BIG_USHORT = struct.Struct('>H')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
233 STRUCT_BIG_ULONG = struct.Struct('>L')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
234 STRUCT_BIG_ULONGLONG = struct.Struct('>Q')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
235
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
236 SPECIAL_NONE = 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
237 SPECIAL_START_INDEFINITE_BYTESTRING = 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
238 SPECIAL_START_ARRAY = 2
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
239 SPECIAL_START_MAP = 3
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
240 SPECIAL_START_SET = 4
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
241 SPECIAL_INDEFINITE_BREAK = 5
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
242
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
243 def decodeitem(b, offset=0):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
244 """Decode a new CBOR value from a buffer at offset.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
245
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
246 This function attempts to decode up to one complete CBOR value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
247 from ``b`` starting at offset ``offset``.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
248
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
249 The beginning of a collection (such as an array, map, set, or
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
250 indefinite length bytestring) counts as a single value. For these
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
251 special cases, a state flag will indicate that a special value was seen.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
252
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
253 When called, the function either returns a decoded value or gives
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
254 a hint as to how many more bytes are needed to do so. By calling
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
255 the function repeatedly given a stream of bytes, the caller can
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
256 build up the original values.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
257
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
258 Returns a tuple with the following elements:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
259
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
260 * Bool indicating whether a complete value was decoded.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
261 * A decoded value if first value is True otherwise None
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
262 * Integer number of bytes. If positive, the number of bytes
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
263 read. If negative, the number of bytes we need to read to
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
264 decode this value or the next chunk in this value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
265 * One of the ``SPECIAL_*`` constants indicating special treatment
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
266 for this value. ``SPECIAL_NONE`` means this is a fully decoded
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
267 simple value (such as an integer or bool).
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
268 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
269
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
270 initial = _elementtointeger(b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
271 offset += 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
272
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
273 majortype = initial >> 5
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
274 subtype = initial & SUBTYPE_MASK
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
275
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
276 if majortype == MAJOR_TYPE_UINT:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
277 complete, value, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
278
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
279 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
280 return True, value, readcount + 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
281 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
282 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
283
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
284 elif majortype == MAJOR_TYPE_NEGINT:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
285 # Negative integers are the same as UINT except inverted minus 1.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
286 complete, value, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
287
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
288 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
289 return True, -value - 1, readcount + 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
290 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
291 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
292
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
293 elif majortype == MAJOR_TYPE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
294 # Beginning of bytestrings are treated as uints in order to
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
295 # decode their length, which may be indefinite.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
296 complete, size, readcount = decodeuint(subtype, b, offset,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
297 allowindefinite=True)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
298
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
299 # We don't know the size of the bytestring. It must be a definitive
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
300 # length since the indefinite subtype would be encoded in the initial
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
301 # byte.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
302 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
303 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
304
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
305 # We know the length of the bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
306 if size is not None:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
307 # And the data is available in the buffer.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
308 if offset + readcount + size <= len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
309 value = b[offset + readcount:offset + readcount + size]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
310 return True, value, readcount + size + 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
311
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
312 # And we need more data in order to return the bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
313 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
314 wanted = len(b) - offset - readcount - size
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
315 return False, None, wanted, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
316
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
317 # It is an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
318 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
319 return True, None, 1, SPECIAL_START_INDEFINITE_BYTESTRING
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
320
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
321 elif majortype == MAJOR_TYPE_STRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
322 raise CBORDecodeError('string major type not supported')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
323
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
324 elif majortype == MAJOR_TYPE_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
325 # Beginning of arrays are treated as uints in order to decode their
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
326 # length. We don't allow indefinite length arrays.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
327 complete, size, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
328
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
329 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
330 return True, size, readcount + 1, SPECIAL_START_ARRAY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
331 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
332 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
333
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
334 elif majortype == MAJOR_TYPE_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
335 # Beginning of maps are treated as uints in order to decode their
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
336 # number of elements. We don't allow indefinite length arrays.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
337 complete, size, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
338
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
339 if complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
340 return True, size, readcount + 1, SPECIAL_START_MAP
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
341 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
342 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
343
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
344 elif majortype == MAJOR_TYPE_SEMANTIC:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
345 # Semantic tag value is read the same as a uint.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
346 complete, tagvalue, readcount = decodeuint(subtype, b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
347
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
348 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
349 return False, None, readcount, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
350
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
351 # This behavior here is a little wonky. The main type being "decorated"
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
352 # by this semantic tag follows. A more robust parser would probably emit
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
353 # a special flag indicating this as a semantic tag and let the caller
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
354 # deal with the types that follow. But since we don't support many
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
355 # semantic tags, it is easier to deal with the special cases here and
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
356 # hide complexity from the caller. If we add support for more semantic
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
357 # tags, we should probably move semantic tag handling into the caller.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
358 if tagvalue == SEMANTIC_TAG_FINITE_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
359 if offset + readcount >= len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
360 return False, None, -1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
361
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
362 complete, size, readcount2, special = decodeitem(b,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
363 offset + readcount)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
364
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
365 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
366 return False, None, readcount2, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
367
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
368 if special != SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
369 raise CBORDecodeError('expected array after finite set '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
370 'semantic tag')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
371
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
372 return True, size, readcount + readcount2 + 1, SPECIAL_START_SET
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
373
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
374 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
375 raise CBORDecodeError('semantic tag %d not allowed' % tagvalue)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
376
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
377 elif majortype == MAJOR_TYPE_SPECIAL:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
378 # Only specific values for the information field are allowed.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
379 if subtype == SUBTYPE_FALSE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
380 return True, False, 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
381 elif subtype == SUBTYPE_TRUE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
382 return True, True, 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
383 elif subtype == SUBTYPE_NULL:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
384 return True, None, 1, SPECIAL_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
385 elif subtype == SUBTYPE_INDEFINITE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
386 return True, None, 1, SPECIAL_INDEFINITE_BREAK
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
387 # If value is 24, subtype is in next byte.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
388 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
389 raise CBORDecodeError('special type %d not allowed' % subtype)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
390 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
391 assert False
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
392
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
393 def decodeuint(subtype, b, offset=0, allowindefinite=False):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
394 """Decode an unsigned integer.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
395
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
396 ``subtype`` is the lower 5 bits from the initial byte CBOR item
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
397 "header." ``b`` is a buffer containing bytes. ``offset`` points to
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
398 the index of the first byte after the byte that ``subtype`` was
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
399 derived from.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
400
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
401 ``allowindefinite`` allows the special indefinite length value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
402 indicator.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
403
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
404 Returns a 3-tuple of (successful, value, count).
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
405
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
406 The first element is a bool indicating if decoding completed. The 2nd
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
407 is the decoded integer value or None if not fully decoded or the subtype
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
408 is 31 and ``allowindefinite`` is True. The 3rd value is the count of bytes.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
409 If positive, it is the number of additional bytes decoded. If negative,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
410 it is the number of additional bytes needed to decode this value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
411 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
412
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
413 # Small values are inline.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
414 if subtype < 24:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
415 return True, subtype, 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
416 # Indefinite length specifier.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
417 elif subtype == 31:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
418 if allowindefinite:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
419 return True, None, 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
420 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
421 raise CBORDecodeError('indefinite length uint not allowed here')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
422 elif subtype >= 28:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
423 raise CBORDecodeError('unsupported subtype on integer type: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
424 subtype)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
425
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
426 if subtype == 24:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
427 s = STRUCT_BIG_UBYTE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
428 elif subtype == 25:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
429 s = STRUCT_BIG_USHORT
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
430 elif subtype == 26:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
431 s = STRUCT_BIG_ULONG
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
432 elif subtype == 27:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
433 s = STRUCT_BIG_ULONGLONG
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
434 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
435 raise CBORDecodeError('bounds condition checking violation')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
436
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
437 if len(b) - offset >= s.size:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
438 return True, s.unpack_from(b, offset)[0], s.size
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
439 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
440 return False, None, len(b) - offset - s.size
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
441
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
442 class bytestringchunk(bytes):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
443 """Represents a chunk/segment in an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
444
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
445 This behaves like a ``bytes`` but in addition has the ``isfirst``
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
446 and ``islast`` attributes indicating whether this chunk is the first
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
447 or last in an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
448 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
449
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
450 def __new__(cls, v, first=False, last=False):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
451 self = bytes.__new__(cls, v)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
452 self.isfirst = first
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
453 self.islast = last
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
454
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
455 return self
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
456
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
457 class sansiodecoder(object):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
458 """A CBOR decoder that doesn't perform its own I/O.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
459
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
460 To use, construct an instance and feed it segments containing
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
461 CBOR-encoded bytes via ``decode()``. The return value from ``decode()``
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
462 indicates whether a fully-decoded value is available, how many bytes
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
463 were consumed, and offers a hint as to how many bytes should be fed
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
464 in next time to decode the next value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
465
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
466 The decoder assumes it will decode N discrete CBOR values, not just
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
467 a single value. i.e. if the bytestream contains uints packed one after
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
468 the other, the decoder will decode them all, rather than just the initial
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
469 one.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
470
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
471 When ``decode()`` indicates a value is available, call ``getavailable()``
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
472 to return all fully decoded values.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
473
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
474 ``decode()`` can partially decode input. It is up to the caller to keep
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
475 track of what data was consumed and to pass unconsumed data in on the
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
476 next invocation.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
477
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
478 The decoder decodes atomically at the *item* level. See ``decodeitem()``.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
479 If an *item* cannot be fully decoded, the decoder won't record it as
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
480 partially consumed. Instead, the caller will be instructed to pass in
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
481 the initial bytes of this item on the next invocation. This does result
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
482 in some redundant parsing. But the overhead should be minimal.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
483
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
484 This decoder only supports a subset of CBOR as required by Mercurial.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
485 It lacks support for:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
486
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
487 * Indefinite length arrays
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
488 * Indefinite length maps
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
489 * Use of indefinite length bytestrings as keys or values within
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
490 arrays, maps, or sets.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
491 * Nested arrays, maps, or sets within sets
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
492 * Any semantic tag that isn't a mathematical finite set
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
493 * Floating point numbers
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
494 * Undefined special value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
495
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
496 CBOR types are decoded to Python types as follows:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
497
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
498 uint -> int
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
499 negint -> int
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
500 bytestring -> bytes
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
501 map -> dict
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
502 array -> list
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
503 True -> bool
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
504 False -> bool
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
505 null -> None
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
506 indefinite length bytestring chunk -> [bytestringchunk]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
507
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
508 The only non-obvious mapping here is an indefinite length bytestring
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
509 to the ``bytestringchunk`` type. This is to facilitate streaming
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
510 indefinite length bytestrings out of the decoder and to differentiate
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
511 a regular bytestring from an indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
512 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
513
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
514 _STATE_NONE = 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
515 _STATE_WANT_MAP_KEY = 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
516 _STATE_WANT_MAP_VALUE = 2
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
517 _STATE_WANT_ARRAY_VALUE = 3
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
518 _STATE_WANT_SET_VALUE = 4
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
519 _STATE_WANT_BYTESTRING_CHUNK_FIRST = 5
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
520 _STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT = 6
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
521
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
522 def __init__(self):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
523 # TODO add support for limiting size of bytestrings
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
524 # TODO add support for limiting number of keys / values in collections
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
525 # TODO add support for limiting size of buffered partial values
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
526
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
527 self.decodedbytecount = 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
528
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
529 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
530
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
531 # Stack of active nested collections. Each entry is a dict describing
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
532 # the collection.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
533 self._collectionstack = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
534
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
535 # Fully decoded key to use for the current map.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
536 self._currentmapkey = None
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
537
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
538 # Fully decoded values available for retrieval.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
539 self._decodedvalues = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
540
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
541 @property
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
542 def inprogress(self):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
543 """Whether the decoder has partially decoded a value."""
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
544 return self._state != self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
545
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
546 def decode(self, b, offset=0):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
547 """Attempt to decode bytes from an input buffer.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
548
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
549 ``b`` is a collection of bytes and ``offset`` is the byte
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
550 offset within that buffer from which to begin reading data.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
551
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
552 ``b`` must support ``len()`` and accessing bytes slices via
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
553 ``__slice__``. Typically ``bytes`` instances are used.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
554
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
555 Returns a tuple with the following fields:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
556
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
557 * Bool indicating whether values are available for retrieval.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
558 * Integer indicating the number of bytes that were fully consumed,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
559 starting from ``offset``.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
560 * Integer indicating the number of bytes that are desired for the
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
561 next call in order to decode an item.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
562 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
563 if not b:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
564 return bool(self._decodedvalues), 0, 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
565
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
566 initialoffset = offset
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
567
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
568 # We could easily split the body of this loop into a function. But
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
569 # Python performance is sensitive to function calls and collections
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
570 # are composed of many items. So leaving as a while loop could help
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
571 # with performance. One thing that may not help is the use of
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
572 # if..elif versus a lookup/dispatch table. There may be value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
573 # in switching that.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
574 while offset < len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
575 # Attempt to decode an item. This could be a whole value or a
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
576 # special value indicating an event, such as start or end of a
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
577 # collection or indefinite length type.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
578 complete, value, readcount, special = decodeitem(b, offset)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
579
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
580 if readcount > 0:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
581 self.decodedbytecount += readcount
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
582
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
583 if not complete:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
584 assert readcount < 0
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
585 return (
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
586 bool(self._decodedvalues),
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
587 offset - initialoffset,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
588 -readcount,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
589 )
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
590
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
591 offset += readcount
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
592
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
593 # No nested state. We either have a full value or beginning of a
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
594 # complex value to deal with.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
595 if self._state == self._STATE_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
596 # A normal value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
597 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
598 self._decodedvalues.append(value)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
599
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
600 elif special == SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
601 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
602 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
603 'v': [],
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
604 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
605 self._state = self._STATE_WANT_ARRAY_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
606
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
607 elif special == SPECIAL_START_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
608 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
609 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
610 'v': {},
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
611 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
612 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
613
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
614 elif special == SPECIAL_START_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
615 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
616 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
617 'v': set(),
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
618 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
619 self._state = self._STATE_WANT_SET_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
620
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
621 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
622 self._state = self._STATE_WANT_BYTESTRING_CHUNK_FIRST
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
623
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
624 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
625 raise CBORDecodeError('unhandled special state: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
626 special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
627
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
628 # This value becomes an element of the current array.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
629 elif self._state == self._STATE_WANT_ARRAY_VALUE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
630 # Simple values get appended.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
631 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
632 c = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
633 c['v'].append(value)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
634 c['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
635
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
636 # self._state doesn't need changed.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
637
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
638 # An array nested within an array.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
639 elif special == SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
640 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
641 newvalue = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
642
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
643 lastc['v'].append(newvalue)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
644 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
645
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
646 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
647 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
648 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
649 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
650
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
651 # self._state doesn't need changed.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
652
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
653 # A map nested within an array.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
654 elif special == SPECIAL_START_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
655 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
656 newvalue = {}
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
657
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
658 lastc['v'].append(newvalue)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
659 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
660
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
661 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
662 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
663 'v': newvalue
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
664 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
665
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
666 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
667
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
668 elif special == SPECIAL_START_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
669 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
670 newvalue = set()
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
671
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
672 lastc['v'].append(newvalue)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
673 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
674
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
675 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
676 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
677 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
678 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
679
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
680 self._state = self._STATE_WANT_SET_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
681
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
682 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
683 raise CBORDecodeError('indefinite length bytestrings '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
684 'not allowed as array values')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
685
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
686 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
687 raise CBORDecodeError('unhandled special item when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
688 'expecting array value: %d' % special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
689
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
690 # This value becomes the key of the current map instance.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
691 elif self._state == self._STATE_WANT_MAP_KEY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
692 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
693 self._currentmapkey = value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
694 self._state = self._STATE_WANT_MAP_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
695
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
696 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
697 raise CBORDecodeError('indefinite length bytestrings '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
698 'not allowed as map keys')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
699
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
700 elif special in (SPECIAL_START_ARRAY, SPECIAL_START_MAP,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
701 SPECIAL_START_SET):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
702 raise CBORDecodeError('collections not supported as map '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
703 'keys')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
704
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
705 # We do not allow special values to be used as map keys.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
706 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
707 raise CBORDecodeError('unhandled special item when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
708 'expecting map key: %d' % special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
709
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
710 # This value becomes the value of the current map key.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
711 elif self._state == self._STATE_WANT_MAP_VALUE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
712 # Simple values simply get inserted into the map.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
713 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
714 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
715 lastc['v'][self._currentmapkey] = value
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
716 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
717
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
718 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
719
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
720 # A new array is used as the map value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
721 elif special == SPECIAL_START_ARRAY:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
722 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
723 newvalue = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
724
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
725 lastc['v'][self._currentmapkey] = newvalue
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
726 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
727
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
728 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
729 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
730 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
731 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
732
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
733 self._state = self._STATE_WANT_ARRAY_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
734
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
735 # A new map is used as the map value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
736 elif special == SPECIAL_START_MAP:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
737 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
738 newvalue = {}
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
739
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
740 lastc['v'][self._currentmapkey] = newvalue
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
741 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
742
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
743 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
744 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
745 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
746 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
747
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
748 self._state = self._STATE_WANT_MAP_KEY
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
749
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
750 # A new set is used as the map value.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
751 elif special == SPECIAL_START_SET:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
752 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
753 newvalue = set()
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
754
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
755 lastc['v'][self._currentmapkey] = newvalue
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
756 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
757
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
758 self._collectionstack.append({
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
759 'remaining': value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
760 'v': newvalue,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
761 })
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
762
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
763 self._state = self._STATE_WANT_SET_VALUE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
764
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
765 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
766 raise CBORDecodeError('indefinite length bytestrings not '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
767 'allowed as map values')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
768
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
769 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
770 raise CBORDecodeError('unhandled special item when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
771 'expecting map value: %d' % special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
772
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
773 self._currentmapkey = None
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
774
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
775 # This value is added to the current set.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
776 elif self._state == self._STATE_WANT_SET_VALUE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
777 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
778 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
779 lastc['v'].add(value)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
780 lastc['remaining'] -= 1
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
781
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
782 elif special == SPECIAL_START_INDEFINITE_BYTESTRING:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
783 raise CBORDecodeError('indefinite length bytestrings not '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
784 'allowed as set values')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
785
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
786 elif special in (SPECIAL_START_ARRAY,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
787 SPECIAL_START_MAP,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
788 SPECIAL_START_SET):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
789 raise CBORDecodeError('collections not allowed as set '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
790 'values')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
791
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
792 # We don't allow non-trivial types to exist as set values.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
793 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
794 raise CBORDecodeError('unhandled special item when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
795 'expecting set value: %d' % special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
796
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
797 # This value represents the first chunk in an indefinite length
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
798 # bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
799 elif self._state == self._STATE_WANT_BYTESTRING_CHUNK_FIRST:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
800 # We received a full chunk.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
801 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
802 self._decodedvalues.append(bytestringchunk(value,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
803 first=True))
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
804
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
805 self._state = self._STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
806
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
807 # The end of stream marker. This means it is an empty
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
808 # indefinite length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
809 elif special == SPECIAL_INDEFINITE_BREAK:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
810 # We /could/ convert this to a b''. But we want to preserve
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
811 # the nature of the underlying data so consumers expecting
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
812 # an indefinite length bytestring get one.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
813 self._decodedvalues.append(bytestringchunk(b'',
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
814 first=True,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
815 last=True))
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
816
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
817 # Since indefinite length bytestrings can't be used in
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
818 # collections, we must be at the root level.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
819 assert not self._collectionstack
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
820 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
821
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
822 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
823 raise CBORDecodeError('unexpected special value when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
824 'expecting bytestring chunk: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
825 special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
826
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
827 # This value represents the non-initial chunk in an indefinite
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
828 # length bytestring.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
829 elif self._state == self._STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
830 # We received a full chunk.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
831 if special == SPECIAL_NONE:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
832 self._decodedvalues.append(bytestringchunk(value))
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
833
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
834 # The end of stream marker.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
835 elif special == SPECIAL_INDEFINITE_BREAK:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
836 self._decodedvalues.append(bytestringchunk(b'', last=True))
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
837
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
838 # Since indefinite length bytestrings can't be used in
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
839 # collections, we must be at the root level.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
840 assert not self._collectionstack
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
841 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
842
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
843 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
844 raise CBORDecodeError('unexpected special value when '
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
845 'expecting bytestring chunk: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
846 special)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
847
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
848 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
849 raise CBORDecodeError('unhandled decoder state: %d' %
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
850 self._state)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
851
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
852 # We could have just added the final value in a collection. End
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
853 # all complete collections at the top of the stack.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
854 while True:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
855 # Bail if we're not waiting on a new collection item.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
856 if self._state not in (self._STATE_WANT_ARRAY_VALUE,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
857 self._STATE_WANT_MAP_KEY,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
858 self._STATE_WANT_SET_VALUE):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
859 break
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
860
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
861 # Or we are expecting more items for this collection.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
862 lastc = self._collectionstack[-1]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
863
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
864 if lastc['remaining']:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
865 break
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
866
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
867 # The collection at the top of the stack is complete.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
868
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
869 # Discard it, as it isn't needed for future items.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
870 self._collectionstack.pop()
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
871
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
872 # If this is a nested collection, we don't emit it, since it
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
873 # will be emitted by its parent collection. But we do need to
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
874 # update state to reflect what the new top-most collection
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
875 # on the stack is.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
876 if self._collectionstack:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
877 self._state = {
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
878 list: self._STATE_WANT_ARRAY_VALUE,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
879 dict: self._STATE_WANT_MAP_KEY,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
880 set: self._STATE_WANT_SET_VALUE,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
881 }[type(self._collectionstack[-1]['v'])]
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
882
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
883 # If this is the root collection, emit it.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
884 else:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
885 self._decodedvalues.append(lastc['v'])
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
886 self._state = self._STATE_NONE
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
887
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
888 return (
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
889 bool(self._decodedvalues),
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
890 offset - initialoffset,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
891 0,
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
892 )
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
893
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
894 def getavailable(self):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
895 """Returns an iterator over fully decoded values.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
896
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
897 Once values are retrieved, they won't be available on the next call.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
898 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
899
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
900 l = list(self._decodedvalues)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
901 self._decodedvalues = []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
902 return l
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
903
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
904 class bufferingdecoder(object):
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
905 """A CBOR decoder that buffers undecoded input.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
906
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
907 This is a glorified wrapper around ``sansiodecoder`` that adds a buffering
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
908 layer. All input that isn't consumed by ``sansiodecoder`` will be buffered
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
909 and concatenated with any new input that arrives later.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
910
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
911 TODO consider adding limits as to the maximum amount of data that can
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
912 be buffered.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
913 """
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
914 def __init__(self):
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
915 self._decoder = sansiodecoder()
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
916 self._chunks = []
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
917 self._wanted = 0
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
918
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
919 def decode(self, b):
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
920 """Attempt to decode bytes to CBOR values.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
921
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
922 Returns a tuple with the following fields:
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
923
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
924 * Bool indicating whether new values are available for retrieval.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
925 * Integer number of bytes decoded from the new input.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
926 * Integer number of bytes wanted to decode the next value.
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
927 """
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
928 # Our strategy for buffering is to aggregate the incoming chunks in a
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
929 # list until we've received enough data to decode the next item.
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
930 # This is slightly more complicated than using an ``io.BytesIO``
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
931 # or continuously concatenating incoming data. However, because it
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
932 # isn't constantly reallocating backing memory for a growing buffer,
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
933 # it prevents excessive memory thrashing and is significantly faster,
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
934 # especially in cases where the percentage of input chunks that don't
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
935 # decode into a full item is high.
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
936
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
937 if self._chunks:
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
938 # A previous call said we needed N bytes to decode the next item.
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
939 # But this call doesn't provide enough data. We buffer the incoming
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
940 # chunk without attempting to decode.
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
941 if len(b) < self._wanted:
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
942 self._chunks.append(b)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
943 self._wanted -= len(b)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
944 return False, 0, self._wanted
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
945
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
946 # Else we may have enough data to decode the next item. Aggregate
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
947 # old data with new and reset the buffer.
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
948 newlen = len(b)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
949 self._chunks.append(b)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
950 b = b''.join(self._chunks)
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
951 self._chunks = []
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
952 oldlen = len(b) - newlen
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
953
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
954 else:
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
955 oldlen = 0
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
956
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
957 available, readcount, wanted = self._decoder.decode(b)
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
958 self._wanted = wanted
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
959
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
960 if readcount < len(b):
40030
62160d3077cd cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39456
diff changeset
961 self._chunks.append(b[readcount:])
39413
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
962
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
963 return available, readcount - oldlen, wanted
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
964
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
965 def getavailable(self):
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
966 return self._decoder.getavailable()
babad5ebaf0a cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39412
diff changeset
967
39411
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
968 def decodeall(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
969 """Decode all CBOR items present in an iterable of bytes.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
970
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
971 In addition to regular decode errors, raises CBORDecodeError if the
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
972 entirety of the passed buffer does not fully decode to complete CBOR
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
973 values. This includes failure to decode any value, incomplete collection
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
974 types, incomplete indefinite length items, and extra data at the end of
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
975 the buffer.
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
976 """
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
977 if not b:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
978 return []
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
979
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
980 decoder = sansiodecoder()
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
981
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
982 havevalues, readcount, wantbytes = decoder.decode(b)
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
983
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
984 if readcount != len(b):
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
985 raise CBORDecodeError('input data not fully consumed')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
986
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
987 if decoder.inprogress:
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
988 raise CBORDecodeError('input data not complete')
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
989
aeb551a3bb8a cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37898
diff changeset
990 return decoder.getavailable()