Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/httpclient/_readers.py @ 17424:e7cfe3587ea4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fix trivial spelling errors
author Mads Kiilerich <mads@kiilerich.com>
date Wed, 15 Aug 2012 22:38:42 +0200
parents 24dbef11f477
children 19d1cc30e7a3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
1 # Copyright 2011, Google Inc.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
2 # All rights reserved.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
3 #
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
4 # Redistribution and use in source and binary forms, with or without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
5 # modification, are permitted provided that the following conditions are
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
6 # met:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
7 #
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
8 # * Redistributions of source code must retain the above copyright
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
9 # notice, this list of conditions and the following disclaimer.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
10 # * Redistributions in binary form must reproduce the above
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
11 # copyright notice, this list of conditions and the following disclaimer
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
12 # in the documentation and/or other materials provided with the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
13 # distribution.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
14 # * Neither the name of Google Inc. nor the names of its
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
15 # contributors may be used to endorse or promote products derived from
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
16 # this software without specific prior written permission.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
17
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
18 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
19 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
20 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
21 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
22 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
23 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
24 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
25 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
26 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
27 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
28 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
29 """Reader objects to abstract out different body response types.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
30
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
31 This module is package-private. It is not expected that these will
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
32 have any clients outside of httpplus.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
33 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
34
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
35 import httplib
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
36 import itertools
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
37 import logging
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
38
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
39 logger = logging.getLogger(__name__)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
40
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
41
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
42 class ReadNotReady(Exception):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
43 """Raised when read() is attempted but not enough data is loaded."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
44
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
45
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
46 class HTTPRemoteClosedError(httplib.HTTPException):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
47 """The server closed the remote socket in the middle of a response."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
48
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
49
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
50 class AbstractReader(object):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
51 """Abstract base class for response readers.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
52
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
53 Subclasses must implement _load, and should implement _close if
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
54 it's not an error for the server to close their socket without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
55 some termination condition being detected during _load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
56 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
57 def __init__(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
58 self._finished = False
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
59 self._done_chunks = []
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
60
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
61 @property
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
62 def available_data(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
63 return sum(map(len, self._done_chunks))
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
64
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
65 def done(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
66 return self._finished
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
67
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
68 def read(self, amt):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
69 if self.available_data < amt and not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
70 raise ReadNotReady()
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
71 need = [amt]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
72 def pred(s):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
73 needed = need[0] > 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
74 need[0] -= len(s)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
75 return needed
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
76 blocks = list(itertools.takewhile(pred, self._done_chunks))
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
77 self._done_chunks = self._done_chunks[len(blocks):]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
78 over_read = sum(map(len, blocks)) - amt
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
79 if over_read > 0 and blocks:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
80 logger.debug('need to reinsert %d data into done chunks', over_read)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
81 last = blocks[-1]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
82 blocks[-1], reinsert = last[:-over_read], last[-over_read:]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
83 self._done_chunks.insert(0, reinsert)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
84 result = ''.join(blocks)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
85 assert len(result) == amt or (self._finished and len(result) < amt)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
86 return result
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
87
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
88 def _load(self, data): # pragma: no cover
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
89 """Subclasses must implement this.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
90
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
91 As data is available to be read out of this object, it should
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
92 be placed into the _done_chunks list. Subclasses should not
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
93 rely on data remaining in _done_chunks forever, as it may be
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
94 reaped if the client is parsing data as it comes in.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
95 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
96 raise NotImplementedError
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
97
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
98 def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
99 """Default implementation of close.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
100
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
101 The default implementation assumes that the reader will mark
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
102 the response as finished on the _finished attribute once the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
103 entire response body has been read. In the event that this is
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
104 not true, the subclass should override the implementation of
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
105 close (for example, close-is-end responses have to set
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
106 self._finished in the close handler.)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
107 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
108 if not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
109 raise HTTPRemoteClosedError(
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
110 'server appears to have closed the socket mid-response')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
111
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
112
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
113 class AbstractSimpleReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
114 """Abstract base class for simple readers that require no response decoding.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
115
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
116 Examples of such responses are Connection: Close (close-is-end)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
117 and responses that specify a content length.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
118 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
119 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
120 if data:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
121 assert not self._finished, (
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
122 'tried to add data (%r) to a closed reader!' % data)
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
123 logger.debug('%s read an additional %d data', self.name, len(data))
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
124 self._done_chunks.append(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
125
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
126
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
127 class CloseIsEndReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
128 """Reader for responses that specify Connection: Close for length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
129 name = 'close-is-end'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
130
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
131 def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
132 logger.info('Marking close-is-end reader as closed.')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
133 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
134
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
135
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
136 class ContentLengthReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
137 """Reader for responses that specify an exact content length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
138 name = 'content-length'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
139
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
140 def __init__(self, amount):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
141 AbstractReader.__init__(self)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
142 self._amount = amount
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
143 if amount == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
144 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
145 self._amount_seen = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
146
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
147 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
148 AbstractSimpleReader._load(self, data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
149 self._amount_seen += len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
150 if self._amount_seen >= self._amount:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
151 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
152 logger.debug('content-length read complete')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
153
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
154
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
155 class ChunkedReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
156 """Reader for chunked transfer encoding responses."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
157 def __init__(self, eol):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
158 AbstractReader.__init__(self)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
159 self._eol = eol
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
160 self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
161 self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
162
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
163 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
164 assert not self._finished, 'tried to add data to a closed reader!'
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
165 logger.debug('chunked read an additional %d data', len(data))
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
166 position = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
167 if self._leftover_data:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
168 logger.debug('chunked reader trying to finish block from leftover data')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
169 # TODO: avoid this string concatenation if possible
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
170 data = self._leftover_data + data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
171 position = self._leftover_skip_amt
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
172 self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
173 self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
174 datalen = len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
175 while position < datalen:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
176 split = data.find(self._eol, position)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
177 if split == -1:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
178 self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
179 self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
180 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
181 amt = int(data[position:split], base=16)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
182 block_start = split + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
183 # If the whole data chunk plus the eol trailer hasn't
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
184 # loaded, we'll wait for the next load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
185 if block_start + amt + len(self._eol) > len(data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
186 self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
187 self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
188 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
189 if amt == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
190 self._finished = True
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
191 logger.debug('closing chunked reader due to chunk of length 0')
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
192 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
193 self._done_chunks.append(data[block_start:block_start + amt])
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
194 position = block_start + amt + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
195 # no-check-code