Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/httpclient/_readers.py @ 18426:01638b51df44

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
hgweb: ignore filtered revision in revnav This changeset checks that a revision is known before adding it to the navigation. This will prevent traceback on filtered repository. This changeset result in an incorrect behaviors, Navigation link may be dropped without any replacement. However this bad navigation generation is much better than a crash
author Pierre-Yves David <pierre-yves.david@logilab.fr>
date Wed, 16 Jan 2013 14:19:28 +0100
parents e7cfe3587ea4
children 19d1cc30e7a3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
1 # Copyright 2011, Google Inc.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
2 # All rights reserved.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
3 #
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
4 # Redistribution and use in source and binary forms, with or without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
5 # modification, are permitted provided that the following conditions are
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
6 # met:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
7 #
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
8 # * Redistributions of source code must retain the above copyright
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
9 # notice, this list of conditions and the following disclaimer.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
10 # * Redistributions in binary form must reproduce the above
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
11 # copyright notice, this list of conditions and the following disclaimer
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
12 # in the documentation and/or other materials provided with the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
13 # distribution.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
14 # * Neither the name of Google Inc. nor the names of its
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
15 # contributors may be used to endorse or promote products derived from
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
16 # this software without specific prior written permission.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
17
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
18 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
19 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
20 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
21 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
22 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
23 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
24 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
25 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
26 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
27 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
28 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
29 """Reader objects to abstract out different body response types.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
30
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
31 This module is package-private. It is not expected that these will
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
32 have any clients outside of httpplus.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
33 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
34
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
35 import httplib
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
36 import itertools
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
37 import logging
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
38
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
39 logger = logging.getLogger(__name__)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
40
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
41
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
42 class ReadNotReady(Exception):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
43 """Raised when read() is attempted but not enough data is loaded."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
44
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
45
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
46 class HTTPRemoteClosedError(httplib.HTTPException):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
47 """The server closed the remote socket in the middle of a response."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
48
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
49
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
50 class AbstractReader(object):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
51 """Abstract base class for response readers.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
52
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
53 Subclasses must implement _load, and should implement _close if
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
54 it's not an error for the server to close their socket without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
55 some termination condition being detected during _load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
56 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
57 def __init__(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
58 self._finished = False
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
59 self._done_chunks = []
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
60
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
61 @property
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
62 def available_data(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
63 return sum(map(len, self._done_chunks))
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
64
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
65 def done(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
66 return self._finished
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
67
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
68 def read(self, amt):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
69 if self.available_data < amt and not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
70 raise ReadNotReady()
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
71 need = [amt]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
72 def pred(s):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
73 needed = need[0] > 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
74 need[0] -= len(s)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
75 return needed
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
76 blocks = list(itertools.takewhile(pred, self._done_chunks))
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
77 self._done_chunks = self._done_chunks[len(blocks):]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
78 over_read = sum(map(len, blocks)) - amt
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
79 if over_read > 0 and blocks:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
80 logger.debug('need to reinsert %d data into done chunks', over_read)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
81 last = blocks[-1]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
82 blocks[-1], reinsert = last[:-over_read], last[-over_read:]
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
83 self._done_chunks.insert(0, reinsert)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
84 result = ''.join(blocks)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
85 assert len(result) == amt or (self._finished and len(result) < amt)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
86 return result
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
87
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
88 def _load(self, data): # pragma: no cover
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
89 """Subclasses must implement this.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
90
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
91 As data is available to be read out of this object, it should
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
92 be placed into the _done_chunks list. Subclasses should not
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
93 rely on data remaining in _done_chunks forever, as it may be
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
94 reaped if the client is parsing data as it comes in.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
95 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
96 raise NotImplementedError
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
97
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
98 def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
99 """Default implementation of close.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
100
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
101 The default implementation assumes that the reader will mark
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
102 the response as finished on the _finished attribute once the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
103 entire response body has been read. In the event that this is
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
104 not true, the subclass should override the implementation of
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
105 close (for example, close-is-end responses have to set
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
106 self._finished in the close handler.)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
107 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
108 if not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
109 raise HTTPRemoteClosedError(
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
110 'server appears to have closed the socket mid-response')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
111
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
112
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
113 class AbstractSimpleReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
114 """Abstract base class for simple readers that require no response decoding.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
115
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
116 Examples of such responses are Connection: Close (close-is-end)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
117 and responses that specify a content length.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
118 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
119 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
120 if data:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
121 assert not self._finished, (
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
122 'tried to add data (%r) to a closed reader!' % data)
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
123 logger.debug('%s read an additional %d data', self.name, len(data))
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
124 self._done_chunks.append(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
125
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
126
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
127 class CloseIsEndReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
128 """Reader for responses that specify Connection: Close for length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
129 name = 'close-is-end'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
130
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
131 def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
132 logger.info('Marking close-is-end reader as closed.')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
133 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
134
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
135
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
136 class ContentLengthReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
137 """Reader for responses that specify an exact content length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
138 name = 'content-length'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
139
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
140 def __init__(self, amount):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
141 AbstractReader.__init__(self)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
142 self._amount = amount
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
143 if amount == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
144 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
145 self._amount_seen = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
146
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
147 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
148 AbstractSimpleReader._load(self, data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
149 self._amount_seen += len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
150 if self._amount_seen >= self._amount:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
151 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
152 logger.debug('content-length read complete')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
153
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
154
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
155 class ChunkedReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
156 """Reader for chunked transfer encoding responses."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
157 def __init__(self, eol):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
158 AbstractReader.__init__(self)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
159 self._eol = eol
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
160 self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
161 self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
162
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
163 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
164 assert not self._finished, 'tried to add data to a closed reader!'
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
165 logger.debug('chunked read an additional %d data', len(data))
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
166 position = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
167 if self._leftover_data:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
168 logger.debug('chunked reader trying to finish block from leftover data')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
169 # TODO: avoid this string concatenation if possible
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
170 data = self._leftover_data + data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
171 position = self._leftover_skip_amt
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
172 self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
173 self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
174 datalen = len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
175 while position < datalen:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
176 split = data.find(self._eol, position)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
177 if split == -1:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
178 self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
179 self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
180 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
181 amt = int(data[position:split], base=16)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
182 block_start = split + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
183 # If the whole data chunk plus the eol trailer hasn't
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
184 # loaded, we'll wait for the next load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
185 if block_start + amt + len(self._eol) > len(data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
186 self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
187 self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
188 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
189 if amt == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
190 self._finished = True
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
191 logger.debug('closing chunked reader due to chunk of length 0')
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
192 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
193 self._done_chunks.append(data[block_start:block_start + amt])
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
194 position = block_start + amt + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
195 # no-check-code