Mercurial > public > mercurial-scm > hg

annotate mercurial/httpclient/_readers.py @ 19038:36733ab7fa05

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
http2: sane readline It turns out that it pays off to read more than a byte at a time with a select in between :)
author Brendan Cully <brendan@kublai.com>
date Fri, 01 Feb 2013 15:00:23 -0800
parents 1fde25ad9396
children fae47ecaa952
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
1 # Copyright 2011, Google Inc.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
2 # All rights reserved.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
3 #
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
4 # Redistribution and use in source and binary forms, with or without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
5 # modification, are permitted provided that the following conditions are
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
6 # met:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
7 #
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
8 # * Redistributions of source code must retain the above copyright
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
9 # notice, this list of conditions and the following disclaimer.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
10 # * Redistributions in binary form must reproduce the above
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
11 # copyright notice, this list of conditions and the following disclaimer
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
12 # in the documentation and/or other materials provided with the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
13 # distribution.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
14 # * Neither the name of Google Inc. nor the names of its
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
15 # contributors may be used to endorse or promote products derived from
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
16 # this software without specific prior written permission.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
17
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
18 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
19 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
20 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
21 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
22 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
23 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
24 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
25 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
26 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
27 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
28 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
29 """Reader objects to abstract out different body response types.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
30
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
31 This module is package-private. It is not expected that these will
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
32 have any clients outside of httpplus.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
33 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
34
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
35 import httplib
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
36 import itertools
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
37 import logging
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
38
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
39 logger = logging.getLogger(__name__)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
40
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
41
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
42 class ReadNotReady(Exception):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
43 """Raised when read() is attempted but not enough data is loaded."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
44
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
45
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
46 class HTTPRemoteClosedError(httplib.HTTPException):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
47 """The server closed the remote socket in the middle of a response."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
48
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
49
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
50 class AbstractReader(object):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
51 """Abstract base class for response readers.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
52
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
53 Subclasses must implement _load, and should implement _close if
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
54 it's not an error for the server to close their socket without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
55 some termination condition being detected during _load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
56 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
57 def __init__(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
58 self._finished = False
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
59 self._done_chunks = []
19036
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
60 self.available_data = 0
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
61
19036
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
62 def addchunk(self, data):
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
63 self._done_chunks.append(data)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
64 self.available_data += len(data)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
65
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
66 def pushchunk(self, data):
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
67 self._done_chunks.insert(0, data)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
68 self.available_data += len(data)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
69
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
70 def popchunk(self):
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
71 b = self._done_chunks.pop(0)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
72 self.available_data -= len(b)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
73
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
74 return b
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
75
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
76 def done(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
77 return self._finished
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
78
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
79 def read(self, amt):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
80 if self.available_data < amt and not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
81 raise ReadNotReady()
19037
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
82 blocks = []
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
83 need = amt
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
84 while self._done_chunks:
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
85 b = self.popchunk()
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
86 if len(b) > need:
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
87 nb = b[:need]
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
88 self.pushchunk(b[need:])
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
89 b = nb
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
90 blocks.append(b)
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
91 need -= len(b)
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
92 if need == 0:
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
93 break
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
94 result = ''.join(blocks)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
95 assert len(result) == amt or (self._finished and len(result) < amt)
19037
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
96
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
97 return result
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
98
19038
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
99 def readto(self, delimstr, blocks = None):
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
100 """return available data chunks up to the first one in which delimstr
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
101 occurs. No data will be returned after delimstr -- the chunk in which
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
102 it occurs will be split and the remainder pushed back onto the available
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
103 data queue. If blocks is supplied chunks will be added to blocks, otherwise
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
104 a new list will be allocated.
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
105 """
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
106 if blocks is None:
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
107 blocks = []
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
108
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
109 while self._done_chunks:
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
110 b = self.popchunk()
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
111 i = b.find(delimstr) + len(delimstr)
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
112 if i:
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
113 if i < len(b):
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
114 self.pushchunk(b[i:])
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
115 blocks.append(b[:i])
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
116 break
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
117 else:
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
118 blocks.append(b)
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
119
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
120 return blocks
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
121
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
122 def _load(self, data): # pragma: no cover
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
123 """Subclasses must implement this.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
124
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
125 As data is available to be read out of this object, it should
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
126 be placed into the _done_chunks list. Subclasses should not
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
127 rely on data remaining in _done_chunks forever, as it may be
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
128 reaped if the client is parsing data as it comes in.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
129 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
130 raise NotImplementedError
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
131
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
132 def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
133 """Default implementation of close.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
134
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
135 The default implementation assumes that the reader will mark
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
136 the response as finished on the _finished attribute once the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
137 entire response body has been read. In the event that this is
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
138 not true, the subclass should override the implementation of
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
139 close (for example, close-is-end responses have to set
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
140 self._finished in the close handler.)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
141 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
142 if not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
143 raise HTTPRemoteClosedError(
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
144 'server appears to have closed the socket mid-response')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
145
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
146
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
147 class AbstractSimpleReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
148 """Abstract base class for simple readers that require no response decoding.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
149
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
150 Examples of such responses are Connection: Close (close-is-end)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
151 and responses that specify a content length.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
152 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
153 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
154 if data:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
155 assert not self._finished, (
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
156 'tried to add data (%r) to a closed reader!' % data)
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
157 logger.debug('%s read an additional %d data', self.name, len(data))
19036
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
158 self.addchunk(data)
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
159
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
160
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
161 class CloseIsEndReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
162 """Reader for responses that specify Connection: Close for length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
163 name = 'close-is-end'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
164
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
165 def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
166 logger.info('Marking close-is-end reader as closed.')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
167 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
168
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
169
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
170 class ContentLengthReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
171 """Reader for responses that specify an exact content length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
172 name = 'content-length'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
173
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
174 def __init__(self, amount):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
175 AbstractReader.__init__(self)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
176 self._amount = amount
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
177 if amount == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
178 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
179 self._amount_seen = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
180
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
181 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
182 AbstractSimpleReader._load(self, data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
183 self._amount_seen += len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
184 if self._amount_seen >= self._amount:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
185 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
186 logger.debug('content-length read complete')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
187
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
188
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
189 class ChunkedReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
190 """Reader for chunked transfer encoding responses."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
191 def __init__(self, eol):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
192 AbstractReader.__init__(self)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
193 self._eol = eol
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
194 self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
195 self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
196
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
197 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
198 assert not self._finished, 'tried to add data to a closed reader!'
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
199 logger.debug('chunked read an additional %d data', len(data))
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
200 position = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
201 if self._leftover_data:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
202 logger.debug('chunked reader trying to finish block from leftover data')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
203 # TODO: avoid this string concatenation if possible
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
204 data = self._leftover_data + data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
205 position = self._leftover_skip_amt
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
206 self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
207 self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
208 datalen = len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
209 while position < datalen:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
210 split = data.find(self._eol, position)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
211 if split == -1:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
212 self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
213 self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
214 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
215 amt = int(data[position:split], base=16)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
216 block_start = split + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
217 # If the whole data chunk plus the eol trailer hasn't
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
218 # loaded, we'll wait for the next load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
219 if block_start + amt + len(self._eol) > len(data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
220 self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
221 self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
222 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
223 if amt == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
224 self._finished = True
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
225 logger.debug('closing chunked reader due to chunk of length 0')
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
226 return
19036
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
227 self.addchunk(data[block_start:block_start + amt])
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
228 position = block_start + amt + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
229 # no-check-code