Mercurial > public > mercurial-scm > hg
annotate mercurial/utils/cborutil.py @ 40030:62160d3077cd
cborutil: change buffering strategy
Profiling revealed that we were spending a lot of time on the
line that was concatenating the old buffer with the incoming data
when attempting to decode long byte strings, such as manifest
revisions.
Essentially, we were feeding N chunks of size len(X) << len(Y) into
decode() and continuously allocating a new, larger buffer to hold
the undecoded input. This created substantial memory churn and
slowed down execution.
Changing the code to aggregate pending chunks in a list until we
have enough data to fully decode the next atom makes things much
more efficient.
I don't have exact data, but I recall the old code spending >1s
on manifest fulltexts from the mozilla-unified repo. The new code
doesn't significantly appear in profile output.
Differential Revision: https://phab.mercurial-scm.org/D4854
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Wed, 03 Oct 2018 09:43:01 -0700 |
parents | 8d858fbf2759 |
children | b638219a23c3 |
rev | line source |
---|---|
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
1 # cborutil.py - CBOR extensions |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
2 # |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com> |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
4 # |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
7 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
8 from __future__ import absolute_import |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
9 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
10 import struct |
39411
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
11 import sys |
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
12 |
39456
8d858fbf2759
cbor: teach the encoder to handle python `long` type for Windows
Matt Harbison <matt_harbison@yahoo.com>
parents:
39413
diff
changeset
|
13 from .. import pycompat |
8d858fbf2759
cbor: teach the encoder to handle python `long` type for Windows
Matt Harbison <matt_harbison@yahoo.com>
parents:
39413
diff
changeset
|
14 |
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
15 # Very short very of RFC 7049... |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
16 # |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
17 # Each item begins with a byte. The 3 high bits of that byte denote the |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
18 # "major type." The lower 5 bits denote the "subtype." Each major type |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
19 # has its own encoding mechanism. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
20 # |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
21 # Most types have lengths. However, bytestring, string, array, and map |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
22 # can be indefinite length. These are denotes by a subtype with value 31. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
23 # Sub-components of those types then come afterwards and are terminated |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
24 # by a "break" byte. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
25 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
26 MAJOR_TYPE_UINT = 0 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
27 MAJOR_TYPE_NEGINT = 1 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
28 MAJOR_TYPE_BYTESTRING = 2 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
29 MAJOR_TYPE_STRING = 3 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
30 MAJOR_TYPE_ARRAY = 4 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
31 MAJOR_TYPE_MAP = 5 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
32 MAJOR_TYPE_SEMANTIC = 6 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
33 MAJOR_TYPE_SPECIAL = 7 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
34 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
35 SUBTYPE_MASK = 0b00011111 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
36 |
39411
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
37 SUBTYPE_FALSE = 20 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
38 SUBTYPE_TRUE = 21 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
39 SUBTYPE_NULL = 22 |
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
40 SUBTYPE_HALF_FLOAT = 25 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
41 SUBTYPE_SINGLE_FLOAT = 26 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
42 SUBTYPE_DOUBLE_FLOAT = 27 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
43 SUBTYPE_INDEFINITE = 31 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
44 |
39411
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
45 SEMANTIC_TAG_FINITE_SET = 258 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
46 |
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
47 # Indefinite types begin with their major type ORd with information value 31. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
48 BEGIN_INDEFINITE_BYTESTRING = struct.pack( |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
49 r'>B', MAJOR_TYPE_BYTESTRING << 5 | SUBTYPE_INDEFINITE) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
50 BEGIN_INDEFINITE_ARRAY = struct.pack( |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
51 r'>B', MAJOR_TYPE_ARRAY << 5 | SUBTYPE_INDEFINITE) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
52 BEGIN_INDEFINITE_MAP = struct.pack( |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
53 r'>B', MAJOR_TYPE_MAP << 5 | SUBTYPE_INDEFINITE) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
54 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
55 ENCODED_LENGTH_1 = struct.Struct(r'>B') |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
56 ENCODED_LENGTH_2 = struct.Struct(r'>BB') |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
57 ENCODED_LENGTH_3 = struct.Struct(r'>BH') |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
58 ENCODED_LENGTH_4 = struct.Struct(r'>BL') |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
59 ENCODED_LENGTH_5 = struct.Struct(r'>BQ') |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
60 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
61 # The break ends an indefinite length item. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
62 BREAK = b'\xff' |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
63 BREAK_INT = 255 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
64 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
65 def encodelength(majortype, length): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
66 """Obtain a value encoding the major type and its length.""" |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
67 if length < 24: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
68 return ENCODED_LENGTH_1.pack(majortype << 5 | length) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
69 elif length < 256: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
70 return ENCODED_LENGTH_2.pack(majortype << 5 | 24, length) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
71 elif length < 65536: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
72 return ENCODED_LENGTH_3.pack(majortype << 5 | 25, length) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
73 elif length < 4294967296: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
74 return ENCODED_LENGTH_4.pack(majortype << 5 | 26, length) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
75 else: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
76 return ENCODED_LENGTH_5.pack(majortype << 5 | 27, length) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
77 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
78 def streamencodebytestring(v): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
79 yield encodelength(MAJOR_TYPE_BYTESTRING, len(v)) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
80 yield v |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
81 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
82 def streamencodebytestringfromiter(it): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
83 """Convert an iterator of chunks to an indefinite bytestring. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
84 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
85 Given an input that is iterable and each element in the iterator is |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
86 representable as bytes, emit an indefinite length bytestring. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
87 """ |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
88 yield BEGIN_INDEFINITE_BYTESTRING |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
89 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
90 for chunk in it: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
91 yield encodelength(MAJOR_TYPE_BYTESTRING, len(chunk)) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
92 yield chunk |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
93 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
94 yield BREAK |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
95 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
96 def streamencodeindefinitebytestring(source, chunksize=65536): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
97 """Given a large source buffer, emit as an indefinite length bytestring. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
98 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
99 This is a generator of chunks constituting the encoded CBOR data. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
100 """ |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
101 yield BEGIN_INDEFINITE_BYTESTRING |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
102 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
103 i = 0 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
104 l = len(source) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
105 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
106 while True: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
107 chunk = source[i:i + chunksize] |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
108 i += len(chunk) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
109 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
110 yield encodelength(MAJOR_TYPE_BYTESTRING, len(chunk)) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
111 yield chunk |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
112 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
113 if i >= l: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
114 break |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
115 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
116 yield BREAK |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
117 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
118 def streamencodeint(v): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
119 if v >= 18446744073709551616 or v < -18446744073709551616: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
120 raise ValueError('big integers not supported') |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
121 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
122 if v >= 0: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
123 yield encodelength(MAJOR_TYPE_UINT, v) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
124 else: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
125 yield encodelength(MAJOR_TYPE_NEGINT, abs(v) - 1) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
126 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
127 def streamencodearray(l): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
128 """Encode a known size iterable to an array.""" |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
129 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
130 yield encodelength(MAJOR_TYPE_ARRAY, len(l)) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
131 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
132 for i in l: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
133 for chunk in streamencode(i): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
134 yield chunk |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
135 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
136 def streamencodearrayfromiter(it): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
137 """Encode an iterator of items to an indefinite length array.""" |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
138 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
139 yield BEGIN_INDEFINITE_ARRAY |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
140 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
141 for i in it: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
142 for chunk in streamencode(i): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
143 yield chunk |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
144 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
145 yield BREAK |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
146 |
37898
2ae6a3134362
cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents:
37711
diff
changeset
|
147 def _mixedtypesortkey(v): |
2ae6a3134362
cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents:
37711
diff
changeset
|
148 return type(v).__name__, v |
2ae6a3134362
cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents:
37711
diff
changeset
|
149 |
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
150 def streamencodeset(s): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
151 # https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml defines |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
152 # semantic tag 258 for finite sets. |
39411
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
153 yield encodelength(MAJOR_TYPE_SEMANTIC, SEMANTIC_TAG_FINITE_SET) |
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
154 |
37898
2ae6a3134362
cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents:
37711
diff
changeset
|
155 for chunk in streamencodearray(sorted(s, key=_mixedtypesortkey)): |
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
156 yield chunk |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
157 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
158 def streamencodemap(d): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
159 """Encode dictionary to a generator. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
160 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
161 Does not supporting indefinite length dictionaries. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
162 """ |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
163 yield encodelength(MAJOR_TYPE_MAP, len(d)) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
164 |
37898
2ae6a3134362
cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents:
37711
diff
changeset
|
165 for key, value in sorted(d.iteritems(), |
2ae6a3134362
cborutil: port to Python 3
Augie Fackler <augie@google.com>
parents:
37711
diff
changeset
|
166 key=lambda x: _mixedtypesortkey(x[0])): |
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
167 for chunk in streamencode(key): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
168 yield chunk |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
169 for chunk in streamencode(value): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
170 yield chunk |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
171 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
172 def streamencodemapfromiter(it): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
173 """Given an iterable of (key, value), encode to an indefinite length map.""" |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
174 yield BEGIN_INDEFINITE_MAP |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
175 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
176 for key, value in it: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
177 for chunk in streamencode(key): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
178 yield chunk |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
179 for chunk in streamencode(value): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
180 yield chunk |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
181 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
182 yield BREAK |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
183 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
184 def streamencodebool(b): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
185 # major type 7, simple value 20 and 21. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
186 yield b'\xf5' if b else b'\xf4' |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
187 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
188 def streamencodenone(v): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
189 # major type 7, simple value 22. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
190 yield b'\xf6' |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
191 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
192 STREAM_ENCODERS = { |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
193 bytes: streamencodebytestring, |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
194 int: streamencodeint, |
39456
8d858fbf2759
cbor: teach the encoder to handle python `long` type for Windows
Matt Harbison <matt_harbison@yahoo.com>
parents:
39413
diff
changeset
|
195 pycompat.long: streamencodeint, |
37711
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
196 list: streamencodearray, |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
197 tuple: streamencodearray, |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
198 dict: streamencodemap, |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
199 set: streamencodeset, |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
200 bool: streamencodebool, |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
201 type(None): streamencodenone, |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
202 } |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
203 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
204 def streamencode(v): |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
205 """Encode a value in a streaming manner. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
206 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
207 Given an input object, encode it to CBOR recursively. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
208 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
209 Returns a generator of CBOR encoded bytes. There is no guarantee |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
210 that each emitted chunk fully decodes to a value or sub-value. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
211 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
212 Encoding is deterministic - unordered collections are sorted. |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
213 """ |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
214 fn = STREAM_ENCODERS.get(v.__class__) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
215 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
216 if not fn: |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
217 raise ValueError('do not know how to encode %s' % type(v)) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
218 |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
219 return fn(v) |
65a23cc8e75b
cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
220 |
39411
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
221 class CBORDecodeError(Exception): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
222 """Represents an error decoding CBOR.""" |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
223 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
224 if sys.version_info.major >= 3: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
225 def _elementtointeger(b, i): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
226 return b[i] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
227 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
228 def _elementtointeger(b, i): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
229 return ord(b[i]) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
230 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
231 STRUCT_BIG_UBYTE = struct.Struct(r'>B') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
232 STRUCT_BIG_USHORT = struct.Struct('>H') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
233 STRUCT_BIG_ULONG = struct.Struct('>L') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
234 STRUCT_BIG_ULONGLONG = struct.Struct('>Q') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
235 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
236 SPECIAL_NONE = 0 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
237 SPECIAL_START_INDEFINITE_BYTESTRING = 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
238 SPECIAL_START_ARRAY = 2 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
239 SPECIAL_START_MAP = 3 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
240 SPECIAL_START_SET = 4 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
241 SPECIAL_INDEFINITE_BREAK = 5 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
242 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
243 def decodeitem(b, offset=0): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
244 """Decode a new CBOR value from a buffer at offset. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
245 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
246 This function attempts to decode up to one complete CBOR value |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
247 from ``b`` starting at offset ``offset``. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
248 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
249 The beginning of a collection (such as an array, map, set, or |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
250 indefinite length bytestring) counts as a single value. For these |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
251 special cases, a state flag will indicate that a special value was seen. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
252 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
253 When called, the function either returns a decoded value or gives |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
254 a hint as to how many more bytes are needed to do so. By calling |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
255 the function repeatedly given a stream of bytes, the caller can |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
256 build up the original values. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
257 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
258 Returns a tuple with the following elements: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
259 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
260 * Bool indicating whether a complete value was decoded. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
261 * A decoded value if first value is True otherwise None |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
262 * Integer number of bytes. If positive, the number of bytes |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
263 read. If negative, the number of bytes we need to read to |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
264 decode this value or the next chunk in this value. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
265 * One of the ``SPECIAL_*`` constants indicating special treatment |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
266 for this value. ``SPECIAL_NONE`` means this is a fully decoded |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
267 simple value (such as an integer or bool). |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
268 """ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
269 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
270 initial = _elementtointeger(b, offset) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
271 offset += 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
272 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
273 majortype = initial >> 5 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
274 subtype = initial & SUBTYPE_MASK |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
275 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
276 if majortype == MAJOR_TYPE_UINT: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
277 complete, value, readcount = decodeuint(subtype, b, offset) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
278 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
279 if complete: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
280 return True, value, readcount + 1, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
281 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
282 return False, None, readcount, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
283 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
284 elif majortype == MAJOR_TYPE_NEGINT: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
285 # Negative integers are the same as UINT except inverted minus 1. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
286 complete, value, readcount = decodeuint(subtype, b, offset) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
287 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
288 if complete: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
289 return True, -value - 1, readcount + 1, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
290 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
291 return False, None, readcount, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
292 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
293 elif majortype == MAJOR_TYPE_BYTESTRING: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
294 # Beginning of bytestrings are treated as uints in order to |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
295 # decode their length, which may be indefinite. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
296 complete, size, readcount = decodeuint(subtype, b, offset, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
297 allowindefinite=True) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
298 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
299 # We don't know the size of the bytestring. It must be a definitive |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
300 # length since the indefinite subtype would be encoded in the initial |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
301 # byte. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
302 if not complete: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
303 return False, None, readcount, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
304 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
305 # We know the length of the bytestring. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
306 if size is not None: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
307 # And the data is available in the buffer. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
308 if offset + readcount + size <= len(b): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
309 value = b[offset + readcount:offset + readcount + size] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
310 return True, value, readcount + size + 1, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
311 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
312 # And we need more data in order to return the bytestring. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
313 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
314 wanted = len(b) - offset - readcount - size |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
315 return False, None, wanted, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
316 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
317 # It is an indefinite length bytestring. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
318 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
319 return True, None, 1, SPECIAL_START_INDEFINITE_BYTESTRING |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
320 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
321 elif majortype == MAJOR_TYPE_STRING: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
322 raise CBORDecodeError('string major type not supported') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
323 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
324 elif majortype == MAJOR_TYPE_ARRAY: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
325 # Beginning of arrays are treated as uints in order to decode their |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
326 # length. We don't allow indefinite length arrays. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
327 complete, size, readcount = decodeuint(subtype, b, offset) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
328 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
329 if complete: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
330 return True, size, readcount + 1, SPECIAL_START_ARRAY |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
331 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
332 return False, None, readcount, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
333 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
334 elif majortype == MAJOR_TYPE_MAP: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
335 # Beginning of maps are treated as uints in order to decode their |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
336 # number of elements. We don't allow indefinite length arrays. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
337 complete, size, readcount = decodeuint(subtype, b, offset) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
338 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
339 if complete: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
340 return True, size, readcount + 1, SPECIAL_START_MAP |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
341 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
342 return False, None, readcount, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
343 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
344 elif majortype == MAJOR_TYPE_SEMANTIC: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
345 # Semantic tag value is read the same as a uint. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
346 complete, tagvalue, readcount = decodeuint(subtype, b, offset) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
347 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
348 if not complete: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
349 return False, None, readcount, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
350 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
351 # This behavior here is a little wonky. The main type being "decorated" |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
352 # by this semantic tag follows. A more robust parser would probably emit |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
353 # a special flag indicating this as a semantic tag and let the caller |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
354 # deal with the types that follow. But since we don't support many |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
355 # semantic tags, it is easier to deal with the special cases here and |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
356 # hide complexity from the caller. If we add support for more semantic |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
357 # tags, we should probably move semantic tag handling into the caller. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
358 if tagvalue == SEMANTIC_TAG_FINITE_SET: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
359 if offset + readcount >= len(b): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
360 return False, None, -1, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
361 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
362 complete, size, readcount2, special = decodeitem(b, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
363 offset + readcount) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
364 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
365 if not complete: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
366 return False, None, readcount2, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
367 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
368 if special != SPECIAL_START_ARRAY: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
369 raise CBORDecodeError('expected array after finite set ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
370 'semantic tag') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
371 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
372 return True, size, readcount + readcount2 + 1, SPECIAL_START_SET |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
373 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
374 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
375 raise CBORDecodeError('semantic tag %d not allowed' % tagvalue) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
376 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
377 elif majortype == MAJOR_TYPE_SPECIAL: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
378 # Only specific values for the information field are allowed. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
379 if subtype == SUBTYPE_FALSE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
380 return True, False, 1, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
381 elif subtype == SUBTYPE_TRUE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
382 return True, True, 1, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
383 elif subtype == SUBTYPE_NULL: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
384 return True, None, 1, SPECIAL_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
385 elif subtype == SUBTYPE_INDEFINITE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
386 return True, None, 1, SPECIAL_INDEFINITE_BREAK |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
387 # If value is 24, subtype is in next byte. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
388 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
389 raise CBORDecodeError('special type %d not allowed' % subtype) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
390 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
391 assert False |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
392 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
393 def decodeuint(subtype, b, offset=0, allowindefinite=False): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
394 """Decode an unsigned integer. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
395 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
396 ``subtype`` is the lower 5 bits from the initial byte CBOR item |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
397 "header." ``b`` is a buffer containing bytes. ``offset`` points to |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
398 the index of the first byte after the byte that ``subtype`` was |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
399 derived from. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
400 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
401 ``allowindefinite`` allows the special indefinite length value |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
402 indicator. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
403 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
404 Returns a 3-tuple of (successful, value, count). |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
405 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
406 The first element is a bool indicating if decoding completed. The 2nd |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
407 is the decoded integer value or None if not fully decoded or the subtype |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
408 is 31 and ``allowindefinite`` is True. The 3rd value is the count of bytes. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
409 If positive, it is the number of additional bytes decoded. If negative, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
410 it is the number of additional bytes needed to decode this value. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
411 """ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
412 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
413 # Small values are inline. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
414 if subtype < 24: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
415 return True, subtype, 0 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
416 # Indefinite length specifier. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
417 elif subtype == 31: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
418 if allowindefinite: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
419 return True, None, 0 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
420 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
421 raise CBORDecodeError('indefinite length uint not allowed here') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
422 elif subtype >= 28: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
423 raise CBORDecodeError('unsupported subtype on integer type: %d' % |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
424 subtype) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
425 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
426 if subtype == 24: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
427 s = STRUCT_BIG_UBYTE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
428 elif subtype == 25: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
429 s = STRUCT_BIG_USHORT |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
430 elif subtype == 26: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
431 s = STRUCT_BIG_ULONG |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
432 elif subtype == 27: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
433 s = STRUCT_BIG_ULONGLONG |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
434 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
435 raise CBORDecodeError('bounds condition checking violation') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
436 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
437 if len(b) - offset >= s.size: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
438 return True, s.unpack_from(b, offset)[0], s.size |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
439 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
440 return False, None, len(b) - offset - s.size |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
441 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
442 class bytestringchunk(bytes): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
443 """Represents a chunk/segment in an indefinite length bytestring. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
444 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
445 This behaves like a ``bytes`` but in addition has the ``isfirst`` |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
446 and ``islast`` attributes indicating whether this chunk is the first |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
447 or last in an indefinite length bytestring. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
448 """ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
449 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
450 def __new__(cls, v, first=False, last=False): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
451 self = bytes.__new__(cls, v) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
452 self.isfirst = first |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
453 self.islast = last |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
454 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
455 return self |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
456 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
457 class sansiodecoder(object): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
458 """A CBOR decoder that doesn't perform its own I/O. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
459 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
460 To use, construct an instance and feed it segments containing |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
461 CBOR-encoded bytes via ``decode()``. The return value from ``decode()`` |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
462 indicates whether a fully-decoded value is available, how many bytes |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
463 were consumed, and offers a hint as to how many bytes should be fed |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
464 in next time to decode the next value. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
465 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
466 The decoder assumes it will decode N discrete CBOR values, not just |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
467 a single value. i.e. if the bytestream contains uints packed one after |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
468 the other, the decoder will decode them all, rather than just the initial |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
469 one. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
470 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
471 When ``decode()`` indicates a value is available, call ``getavailable()`` |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
472 to return all fully decoded values. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
473 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
474 ``decode()`` can partially decode input. It is up to the caller to keep |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
475 track of what data was consumed and to pass unconsumed data in on the |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
476 next invocation. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
477 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
478 The decoder decodes atomically at the *item* level. See ``decodeitem()``. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
479 If an *item* cannot be fully decoded, the decoder won't record it as |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
480 partially consumed. Instead, the caller will be instructed to pass in |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
481 the initial bytes of this item on the next invocation. This does result |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
482 in some redundant parsing. But the overhead should be minimal. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
483 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
484 This decoder only supports a subset of CBOR as required by Mercurial. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
485 It lacks support for: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
486 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
487 * Indefinite length arrays |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
488 * Indefinite length maps |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
489 * Use of indefinite length bytestrings as keys or values within |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
490 arrays, maps, or sets. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
491 * Nested arrays, maps, or sets within sets |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
492 * Any semantic tag that isn't a mathematical finite set |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
493 * Floating point numbers |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
494 * Undefined special value |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
495 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
496 CBOR types are decoded to Python types as follows: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
497 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
498 uint -> int |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
499 negint -> int |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
500 bytestring -> bytes |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
501 map -> dict |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
502 array -> list |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
503 True -> bool |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
504 False -> bool |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
505 null -> None |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
506 indefinite length bytestring chunk -> [bytestringchunk] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
507 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
508 The only non-obvious mapping here is an indefinite length bytestring |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
509 to the ``bytestringchunk`` type. This is to facilitate streaming |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
510 indefinite length bytestrings out of the decoder and to differentiate |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
511 a regular bytestring from an indefinite length bytestring. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
512 """ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
513 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
514 _STATE_NONE = 0 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
515 _STATE_WANT_MAP_KEY = 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
516 _STATE_WANT_MAP_VALUE = 2 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
517 _STATE_WANT_ARRAY_VALUE = 3 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
518 _STATE_WANT_SET_VALUE = 4 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
519 _STATE_WANT_BYTESTRING_CHUNK_FIRST = 5 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
520 _STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT = 6 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
521 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
522 def __init__(self): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
523 # TODO add support for limiting size of bytestrings |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
524 # TODO add support for limiting number of keys / values in collections |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
525 # TODO add support for limiting size of buffered partial values |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
526 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
527 self.decodedbytecount = 0 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
528 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
529 self._state = self._STATE_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
530 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
531 # Stack of active nested collections. Each entry is a dict describing |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
532 # the collection. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
533 self._collectionstack = [] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
534 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
535 # Fully decoded key to use for the current map. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
536 self._currentmapkey = None |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
537 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
538 # Fully decoded values available for retrieval. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
539 self._decodedvalues = [] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
540 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
541 @property |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
542 def inprogress(self): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
543 """Whether the decoder has partially decoded a value.""" |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
544 return self._state != self._STATE_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
545 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
546 def decode(self, b, offset=0): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
547 """Attempt to decode bytes from an input buffer. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
548 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
549 ``b`` is a collection of bytes and ``offset`` is the byte |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
550 offset within that buffer from which to begin reading data. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
551 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
552 ``b`` must support ``len()`` and accessing bytes slices via |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
553 ``__slice__``. Typically ``bytes`` instances are used. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
554 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
555 Returns a tuple with the following fields: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
556 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
557 * Bool indicating whether values are available for retrieval. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
558 * Integer indicating the number of bytes that were fully consumed, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
559 starting from ``offset``. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
560 * Integer indicating the number of bytes that are desired for the |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
561 next call in order to decode an item. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
562 """ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
563 if not b: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
564 return bool(self._decodedvalues), 0, 0 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
565 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
566 initialoffset = offset |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
567 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
568 # We could easily split the body of this loop into a function. But |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
569 # Python performance is sensitive to function calls and collections |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
570 # are composed of many items. So leaving as a while loop could help |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
571 # with performance. One thing that may not help is the use of |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
572 # if..elif versus a lookup/dispatch table. There may be value |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
573 # in switching that. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
574 while offset < len(b): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
575 # Attempt to decode an item. This could be a whole value or a |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
576 # special value indicating an event, such as start or end of a |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
577 # collection or indefinite length type. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
578 complete, value, readcount, special = decodeitem(b, offset) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
579 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
580 if readcount > 0: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
581 self.decodedbytecount += readcount |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
582 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
583 if not complete: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
584 assert readcount < 0 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
585 return ( |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
586 bool(self._decodedvalues), |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
587 offset - initialoffset, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
588 -readcount, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
589 ) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
590 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
591 offset += readcount |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
592 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
593 # No nested state. We either have a full value or beginning of a |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
594 # complex value to deal with. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
595 if self._state == self._STATE_NONE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
596 # A normal value. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
597 if special == SPECIAL_NONE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
598 self._decodedvalues.append(value) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
599 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
600 elif special == SPECIAL_START_ARRAY: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
601 self._collectionstack.append({ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
602 'remaining': value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
603 'v': [], |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
604 }) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
605 self._state = self._STATE_WANT_ARRAY_VALUE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
606 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
607 elif special == SPECIAL_START_MAP: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
608 self._collectionstack.append({ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
609 'remaining': value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
610 'v': {}, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
611 }) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
612 self._state = self._STATE_WANT_MAP_KEY |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
613 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
614 elif special == SPECIAL_START_SET: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
615 self._collectionstack.append({ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
616 'remaining': value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
617 'v': set(), |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
618 }) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
619 self._state = self._STATE_WANT_SET_VALUE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
620 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
621 elif special == SPECIAL_START_INDEFINITE_BYTESTRING: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
622 self._state = self._STATE_WANT_BYTESTRING_CHUNK_FIRST |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
623 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
624 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
625 raise CBORDecodeError('unhandled special state: %d' % |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
626 special) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
627 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
628 # This value becomes an element of the current array. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
629 elif self._state == self._STATE_WANT_ARRAY_VALUE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
630 # Simple values get appended. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
631 if special == SPECIAL_NONE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
632 c = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
633 c['v'].append(value) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
634 c['remaining'] -= 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
635 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
636 # self._state doesn't need changed. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
637 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
638 # An array nested within an array. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
639 elif special == SPECIAL_START_ARRAY: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
640 lastc = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
641 newvalue = [] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
642 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
643 lastc['v'].append(newvalue) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
644 lastc['remaining'] -= 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
645 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
646 self._collectionstack.append({ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
647 'remaining': value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
648 'v': newvalue, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
649 }) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
650 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
651 # self._state doesn't need changed. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
652 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
653 # A map nested within an array. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
654 elif special == SPECIAL_START_MAP: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
655 lastc = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
656 newvalue = {} |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
657 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
658 lastc['v'].append(newvalue) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
659 lastc['remaining'] -= 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
660 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
661 self._collectionstack.append({ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
662 'remaining': value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
663 'v': newvalue |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
664 }) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
665 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
666 self._state = self._STATE_WANT_MAP_KEY |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
667 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
668 elif special == SPECIAL_START_SET: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
669 lastc = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
670 newvalue = set() |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
671 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
672 lastc['v'].append(newvalue) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
673 lastc['remaining'] -= 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
674 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
675 self._collectionstack.append({ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
676 'remaining': value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
677 'v': newvalue, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
678 }) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
679 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
680 self._state = self._STATE_WANT_SET_VALUE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
681 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
682 elif special == SPECIAL_START_INDEFINITE_BYTESTRING: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
683 raise CBORDecodeError('indefinite length bytestrings ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
684 'not allowed as array values') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
685 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
686 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
687 raise CBORDecodeError('unhandled special item when ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
688 'expecting array value: %d' % special) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
689 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
690 # This value becomes the key of the current map instance. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
691 elif self._state == self._STATE_WANT_MAP_KEY: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
692 if special == SPECIAL_NONE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
693 self._currentmapkey = value |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
694 self._state = self._STATE_WANT_MAP_VALUE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
695 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
696 elif special == SPECIAL_START_INDEFINITE_BYTESTRING: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
697 raise CBORDecodeError('indefinite length bytestrings ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
698 'not allowed as map keys') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
699 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
700 elif special in (SPECIAL_START_ARRAY, SPECIAL_START_MAP, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
701 SPECIAL_START_SET): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
702 raise CBORDecodeError('collections not supported as map ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
703 'keys') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
704 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
705 # We do not allow special values to be used as map keys. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
706 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
707 raise CBORDecodeError('unhandled special item when ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
708 'expecting map key: %d' % special) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
709 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
710 # This value becomes the value of the current map key. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
711 elif self._state == self._STATE_WANT_MAP_VALUE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
712 # Simple values simply get inserted into the map. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
713 if special == SPECIAL_NONE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
714 lastc = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
715 lastc['v'][self._currentmapkey] = value |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
716 lastc['remaining'] -= 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
717 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
718 self._state = self._STATE_WANT_MAP_KEY |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
719 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
720 # A new array is used as the map value. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
721 elif special == SPECIAL_START_ARRAY: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
722 lastc = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
723 newvalue = [] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
724 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
725 lastc['v'][self._currentmapkey] = newvalue |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
726 lastc['remaining'] -= 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
727 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
728 self._collectionstack.append({ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
729 'remaining': value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
730 'v': newvalue, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
731 }) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
732 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
733 self._state = self._STATE_WANT_ARRAY_VALUE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
734 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
735 # A new map is used as the map value. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
736 elif special == SPECIAL_START_MAP: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
737 lastc = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
738 newvalue = {} |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
739 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
740 lastc['v'][self._currentmapkey] = newvalue |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
741 lastc['remaining'] -= 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
742 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
743 self._collectionstack.append({ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
744 'remaining': value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
745 'v': newvalue, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
746 }) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
747 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
748 self._state = self._STATE_WANT_MAP_KEY |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
749 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
750 # A new set is used as the map value. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
751 elif special == SPECIAL_START_SET: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
752 lastc = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
753 newvalue = set() |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
754 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
755 lastc['v'][self._currentmapkey] = newvalue |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
756 lastc['remaining'] -= 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
757 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
758 self._collectionstack.append({ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
759 'remaining': value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
760 'v': newvalue, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
761 }) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
762 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
763 self._state = self._STATE_WANT_SET_VALUE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
764 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
765 elif special == SPECIAL_START_INDEFINITE_BYTESTRING: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
766 raise CBORDecodeError('indefinite length bytestrings not ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
767 'allowed as map values') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
768 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
769 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
770 raise CBORDecodeError('unhandled special item when ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
771 'expecting map value: %d' % special) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
772 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
773 self._currentmapkey = None |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
774 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
775 # This value is added to the current set. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
776 elif self._state == self._STATE_WANT_SET_VALUE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
777 if special == SPECIAL_NONE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
778 lastc = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
779 lastc['v'].add(value) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
780 lastc['remaining'] -= 1 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
781 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
782 elif special == SPECIAL_START_INDEFINITE_BYTESTRING: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
783 raise CBORDecodeError('indefinite length bytestrings not ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
784 'allowed as set values') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
785 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
786 elif special in (SPECIAL_START_ARRAY, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
787 SPECIAL_START_MAP, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
788 SPECIAL_START_SET): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
789 raise CBORDecodeError('collections not allowed as set ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
790 'values') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
791 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
792 # We don't allow non-trivial types to exist as set values. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
793 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
794 raise CBORDecodeError('unhandled special item when ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
795 'expecting set value: %d' % special) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
796 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
797 # This value represents the first chunk in an indefinite length |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
798 # bytestring. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
799 elif self._state == self._STATE_WANT_BYTESTRING_CHUNK_FIRST: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
800 # We received a full chunk. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
801 if special == SPECIAL_NONE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
802 self._decodedvalues.append(bytestringchunk(value, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
803 first=True)) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
804 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
805 self._state = self._STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
806 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
807 # The end of stream marker. This means it is an empty |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
808 # indefinite length bytestring. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
809 elif special == SPECIAL_INDEFINITE_BREAK: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
810 # We /could/ convert this to a b''. But we want to preserve |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
811 # the nature of the underlying data so consumers expecting |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
812 # an indefinite length bytestring get one. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
813 self._decodedvalues.append(bytestringchunk(b'', |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
814 first=True, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
815 last=True)) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
816 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
817 # Since indefinite length bytestrings can't be used in |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
818 # collections, we must be at the root level. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
819 assert not self._collectionstack |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
820 self._state = self._STATE_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
821 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
822 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
823 raise CBORDecodeError('unexpected special value when ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
824 'expecting bytestring chunk: %d' % |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
825 special) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
826 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
827 # This value represents the non-initial chunk in an indefinite |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
828 # length bytestring. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
829 elif self._state == self._STATE_WANT_BYTESTRING_CHUNK_SUBSEQUENT: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
830 # We received a full chunk. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
831 if special == SPECIAL_NONE: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
832 self._decodedvalues.append(bytestringchunk(value)) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
833 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
834 # The end of stream marker. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
835 elif special == SPECIAL_INDEFINITE_BREAK: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
836 self._decodedvalues.append(bytestringchunk(b'', last=True)) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
837 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
838 # Since indefinite length bytestrings can't be used in |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
839 # collections, we must be at the root level. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
840 assert not self._collectionstack |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
841 self._state = self._STATE_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
842 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
843 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
844 raise CBORDecodeError('unexpected special value when ' |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
845 'expecting bytestring chunk: %d' % |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
846 special) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
847 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
848 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
849 raise CBORDecodeError('unhandled decoder state: %d' % |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
850 self._state) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
851 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
852 # We could have just added the final value in a collection. End |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
853 # all complete collections at the top of the stack. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
854 while True: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
855 # Bail if we're not waiting on a new collection item. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
856 if self._state not in (self._STATE_WANT_ARRAY_VALUE, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
857 self._STATE_WANT_MAP_KEY, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
858 self._STATE_WANT_SET_VALUE): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
859 break |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
860 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
861 # Or we are expecting more items for this collection. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
862 lastc = self._collectionstack[-1] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
863 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
864 if lastc['remaining']: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
865 break |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
866 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
867 # The collection at the top of the stack is complete. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
868 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
869 # Discard it, as it isn't needed for future items. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
870 self._collectionstack.pop() |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
871 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
872 # If this is a nested collection, we don't emit it, since it |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
873 # will be emitted by its parent collection. But we do need to |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
874 # update state to reflect what the new top-most collection |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
875 # on the stack is. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
876 if self._collectionstack: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
877 self._state = { |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
878 list: self._STATE_WANT_ARRAY_VALUE, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
879 dict: self._STATE_WANT_MAP_KEY, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
880 set: self._STATE_WANT_SET_VALUE, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
881 }[type(self._collectionstack[-1]['v'])] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
882 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
883 # If this is the root collection, emit it. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
884 else: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
885 self._decodedvalues.append(lastc['v']) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
886 self._state = self._STATE_NONE |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
887 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
888 return ( |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
889 bool(self._decodedvalues), |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
890 offset - initialoffset, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
891 0, |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
892 ) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
893 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
894 def getavailable(self): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
895 """Returns an iterator over fully decoded values. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
896 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
897 Once values are retrieved, they won't be available on the next call. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
898 """ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
899 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
900 l = list(self._decodedvalues) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
901 self._decodedvalues = [] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
902 return l |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
903 |
39413
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
904 class bufferingdecoder(object): |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
905 """A CBOR decoder that buffers undecoded input. |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
906 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
907 This is a glorified wrapper around ``sansiodecoder`` that adds a buffering |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
908 layer. All input that isn't consumed by ``sansiodecoder`` will be buffered |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
909 and concatenated with any new input that arrives later. |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
910 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
911 TODO consider adding limits as to the maximum amount of data that can |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
912 be buffered. |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
913 """ |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
914 def __init__(self): |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
915 self._decoder = sansiodecoder() |
40030
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
916 self._chunks = [] |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
917 self._wanted = 0 |
39413
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
918 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
919 def decode(self, b): |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
920 """Attempt to decode bytes to CBOR values. |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
921 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
922 Returns a tuple with the following fields: |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
923 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
924 * Bool indicating whether new values are available for retrieval. |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
925 * Integer number of bytes decoded from the new input. |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
926 * Integer number of bytes wanted to decode the next value. |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
927 """ |
40030
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
928 # Our strategy for buffering is to aggregate the incoming chunks in a |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
929 # list until we've received enough data to decode the next item. |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
930 # This is slightly more complicated than using an ``io.BytesIO`` |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
931 # or continuously concatenating incoming data. However, because it |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
932 # isn't constantly reallocating backing memory for a growing buffer, |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
933 # it prevents excessive memory thrashing and is significantly faster, |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
934 # especially in cases where the percentage of input chunks that don't |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
935 # decode into a full item is high. |
39413
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
936 |
40030
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
937 if self._chunks: |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
938 # A previous call said we needed N bytes to decode the next item. |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
939 # But this call doesn't provide enough data. We buffer the incoming |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
940 # chunk without attempting to decode. |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
941 if len(b) < self._wanted: |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
942 self._chunks.append(b) |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
943 self._wanted -= len(b) |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
944 return False, 0, self._wanted |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
945 |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
946 # Else we may have enough data to decode the next item. Aggregate |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
947 # old data with new and reset the buffer. |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
948 newlen = len(b) |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
949 self._chunks.append(b) |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
950 b = b''.join(self._chunks) |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
951 self._chunks = [] |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
952 oldlen = len(b) - newlen |
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
953 |
39413
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
954 else: |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
955 oldlen = 0 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
956 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
957 available, readcount, wanted = self._decoder.decode(b) |
40030
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
958 self._wanted = wanted |
39413
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
959 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
960 if readcount < len(b): |
40030
62160d3077cd
cborutil: change buffering strategy
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39456
diff
changeset
|
961 self._chunks.append(b[readcount:]) |
39413
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
962 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
963 return available, readcount - oldlen, wanted |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
964 |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
965 def getavailable(self): |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
966 return self._decoder.getavailable() |
babad5ebaf0a
cborutil: add a buffering decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39412
diff
changeset
|
967 |
39411
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
968 def decodeall(b): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
969 """Decode all CBOR items present in an iterable of bytes. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
970 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
971 In addition to regular decode errors, raises CBORDecodeError if the |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
972 entirety of the passed buffer does not fully decode to complete CBOR |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
973 values. This includes failure to decode any value, incomplete collection |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
974 types, incomplete indefinite length items, and extra data at the end of |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
975 the buffer. |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
976 """ |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
977 if not b: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
978 return [] |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
979 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
980 decoder = sansiodecoder() |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
981 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
982 havevalues, readcount, wantbytes = decoder.decode(b) |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
983 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
984 if readcount != len(b): |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
985 raise CBORDecodeError('input data not fully consumed') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
986 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
987 if decoder.inprogress: |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
988 raise CBORDecodeError('input data not complete') |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
989 |
aeb551a3bb8a
cborutil: implement sans I/O decoder
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37898
diff
changeset
|
990 return decoder.getavailable() |