Mercurial > public > mercurial-scm > hg

annotate mercurial/httpclient/_readers.py @ 27601:1ad9da968a2e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
httpclient: update to 938f2107d6e2 of httpplus This enhances proxy support in httpclient a little bit, though I don't know that we used that functionality at all. It also switches httpplus to using absolute_import.
author Augie Fackler <augie@google.com>
date Thu, 31 Dec 2015 13:19:20 -0500
parents fae47ecaa952
children 8a66eda46c98
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
1 # Copyright 2011, Google Inc.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
2 # All rights reserved.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
3 #
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
4 # Redistribution and use in source and binary forms, with or without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
5 # modification, are permitted provided that the following conditions are
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
6 # met:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
7 #
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
8 # * Redistributions of source code must retain the above copyright
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
9 # notice, this list of conditions and the following disclaimer.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
10 # * Redistributions in binary form must reproduce the above
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
11 # copyright notice, this list of conditions and the following disclaimer
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
12 # in the documentation and/or other materials provided with the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
13 # distribution.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
14 # * Neither the name of Google Inc. nor the names of its
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
15 # contributors may be used to endorse or promote products derived from
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
16 # this software without specific prior written permission.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
17
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
18 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
19 # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
20 # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
21 # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
22 # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
23 # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
24 # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
25 # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
26 # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
27 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
28 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
29 """Reader objects to abstract out different body response types.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
30
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
31 This module is package-private. It is not expected that these will
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
32 have any clients outside of httpplus.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
33 """
27601
1ad9da968a2e httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents: 19182
diff changeset
34 from __future__ import absolute_import
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
35
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
36 import httplib
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
37 import logging
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
38
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
39 logger = logging.getLogger(__name__)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
40
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
41
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
42 class ReadNotReady(Exception):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
43 """Raised when read() is attempted but not enough data is loaded."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
44
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
45
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
46 class HTTPRemoteClosedError(httplib.HTTPException):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
47 """The server closed the remote socket in the middle of a response."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
48
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
49
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
50 class AbstractReader(object):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
51 """Abstract base class for response readers.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
52
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
53 Subclasses must implement _load, and should implement _close if
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
54 it's not an error for the server to close their socket without
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
55 some termination condition being detected during _load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
56 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
57 def __init__(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
58 self._finished = False
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
59 self._done_chunks = []
19036
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
60 self.available_data = 0
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
61
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
62 def _addchunk(self, data):
19036
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
63 self._done_chunks.append(data)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
64 self.available_data += len(data)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
65
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
66 def _pushchunk(self, data):
19036
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
67 self._done_chunks.insert(0, data)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
68 self.available_data += len(data)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
69
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
70 def _popchunk(self):
19036
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
71 b = self._done_chunks.pop(0)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
72 self.available_data -= len(b)
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
73
19d1cc30e7a3 http2: track available data as it changes instead of recomputing it
Brendan Cully <brendan@kublai.com>
parents: 17424
diff changeset
74 return b
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
75
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
76 def done(self):
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
77 """Returns true if the response body is entirely read."""
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
78 return self._finished
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
79
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
80 def read(self, amt):
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
81 """Read amt bytes from the response body."""
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
82 if self.available_data < amt and not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
83 raise ReadNotReady()
19037
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
84 blocks = []
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
85 need = amt
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
86 while self._done_chunks:
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
87 b = self._popchunk()
19037
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
88 if len(b) > need:
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
89 nb = b[:need]
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
90 self._pushchunk(b[need:])
19037
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
91 b = nb
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
92 blocks.append(b)
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
93 need -= len(b)
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
94 if need == 0:
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
95 break
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
96 result = ''.join(blocks)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
97 assert len(result) == amt or (self._finished and len(result) < amt)
19037
1fde25ad9396 http2: make read use pushchunk/popchunk, eschew itertools
Brendan Cully <brendan@kublai.com>
parents: 19036
diff changeset
98
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
99 return result
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
100
19038
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
101 def readto(self, delimstr, blocks = None):
27601
1ad9da968a2e httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents: 19182
diff changeset
102 """return available data chunks up to the first one in which
1ad9da968a2e httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents: 19182
diff changeset
103 delimstr occurs. No data will be returned after delimstr --
1ad9da968a2e httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents: 19182
diff changeset
104 the chunk in which it occurs will be split and the remainder
1ad9da968a2e httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents: 19182
diff changeset
105 pushed back onto the available data queue. If blocks is
1ad9da968a2e httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents: 19182
diff changeset
106 supplied chunks will be added to blocks, otherwise a new list
1ad9da968a2e httpclient: update to 938f2107d6e2 of httpplus
Augie Fackler <augie@google.com>
parents: 19182
diff changeset
107 will be allocated.
19038
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
108 """
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
109 if blocks is None:
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
110 blocks = []
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
111
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
112 while self._done_chunks:
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
113 b = self._popchunk()
19038
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
114 i = b.find(delimstr) + len(delimstr)
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
115 if i:
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
116 if i < len(b):
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
117 self._pushchunk(b[i:])
19038
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
118 blocks.append(b[:i])
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
119 break
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
120 else:
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
121 blocks.append(b)
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
122
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
123 return blocks
36733ab7fa05 http2: sane readline
Brendan Cully <brendan@kublai.com>
parents: 19037
diff changeset
124
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
125 def _load(self, data): # pragma: no cover
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
126 """Subclasses must implement this.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
127
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
128 As data is available to be read out of this object, it should
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
129 be placed into the _done_chunks list. Subclasses should not
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
130 rely on data remaining in _done_chunks forever, as it may be
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
131 reaped if the client is parsing data as it comes in.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
132 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
133 raise NotImplementedError
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
134
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
135 def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
136 """Default implementation of close.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
137
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
138 The default implementation assumes that the reader will mark
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
139 the response as finished on the _finished attribute once the
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
140 entire response body has been read. In the event that this is
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
141 not true, the subclass should override the implementation of
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
142 close (for example, close-is-end responses have to set
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
143 self._finished in the close handler.)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
144 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
145 if not self._finished:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
146 raise HTTPRemoteClosedError(
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
147 'server appears to have closed the socket mid-response')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
148
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
149
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
150 class AbstractSimpleReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
151 """Abstract base class for simple readers that require no response decoding.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
152
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
153 Examples of such responses are Connection: Close (close-is-end)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
154 and responses that specify a content length.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
155 """
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
156 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
157 if data:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
158 assert not self._finished, (
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
159 'tried to add data (%r) to a closed reader!' % data)
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
160 logger.debug('%s read an additional %d data',
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
161 self.name, len(data)) # pylint: disable=E1101
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
162 self._addchunk(data)
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
163
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
164
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
165 class CloseIsEndReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
166 """Reader for responses that specify Connection: Close for length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
167 name = 'close-is-end'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
168
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
169 def _close(self):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
170 logger.info('Marking close-is-end reader as closed.')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
171 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
172
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
173
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
174 class ContentLengthReader(AbstractSimpleReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
175 """Reader for responses that specify an exact content length."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
176 name = 'content-length'
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
177
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
178 def __init__(self, amount):
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
179 AbstractSimpleReader.__init__(self)
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
180 self._amount = amount
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
181 if amount == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
182 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
183 self._amount_seen = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
184
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
185 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
186 AbstractSimpleReader._load(self, data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
187 self._amount_seen += len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
188 if self._amount_seen >= self._amount:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
189 self._finished = True
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
190 logger.debug('content-length read complete')
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
191
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
192
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
193 class ChunkedReader(AbstractReader):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
194 """Reader for chunked transfer encoding responses."""
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
195 def __init__(self, eol):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
196 AbstractReader.__init__(self)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
197 self._eol = eol
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
198 self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
199 self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
200
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
201 def _load(self, data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
202 assert not self._finished, 'tried to add data to a closed reader!'
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
203 logger.debug('chunked read an additional %d data', len(data))
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
204 position = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
205 if self._leftover_data:
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
206 logger.debug(
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
207 'chunked reader trying to finish block from leftover data')
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
208 # TODO: avoid this string concatenation if possible
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
209 data = self._leftover_data + data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
210 position = self._leftover_skip_amt
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
211 self._leftover_data = ''
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
212 self._leftover_skip_amt = 0
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
213 datalen = len(data)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
214 while position < datalen:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
215 split = data.find(self._eol, position)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
216 if split == -1:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
217 self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
218 self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
219 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
220 amt = int(data[position:split], base=16)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
221 block_start = split + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
222 # If the whole data chunk plus the eol trailer hasn't
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
223 # loaded, we'll wait for the next load.
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
224 if block_start + amt + len(self._eol) > len(data):
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
225 self._leftover_data = data
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
226 self._leftover_skip_amt = position
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
227 return
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
228 if amt == 0:
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
229 self._finished = True
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 16643
diff changeset
230 logger.debug('closing chunked reader due to chunk of length 0')
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
231 return
19182
fae47ecaa952 httpclient: upgrade to fe8c09e4db64 of httpplus
Augie Fackler <raf@durin42.com>
parents: 19038
diff changeset
232 self._addchunk(data[block_start:block_start + amt])
16643
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
233 position = block_start + amt + len(self._eol)
24dbef11f477 httpclient: update to revision 892730fe7f46 of httpplus
Augie Fackler <raf@durin42.com>
parents:
diff changeset
234 # no-check-code