Mercurial > public > mercurial-scm > hg
annotate mercurial/keepalive.py @ 52640:24ee91ba9aa8
pyupgrade: drop usage of py3 aliases for `OSError`
These were different classes in py2, but now a handful of error classes are just
an alias of `OSError`, like `IOError`, `EnvironmentError`, `WindowsError`, etc.
This is the result of running a hacked version of `pyupgrade` 3.19.1[1]
$ hg files -0 'relglob:**.py' | xargs -0 \
pyupgrade --py38-plus --keep-percent-format --keep-mock --keep-runtime-typing
The hack is because it doesn't have command line switches to disable most
changes, so it makes tons of unrelated changes all at once. The hack is to
1) patch `pyupgrade._main._fix_tokens()` to immediately return its content arg
2) change `pyupgrade._data.register_decorator()` to only register the function
if it's from the fixer we're interested in:
if func.__module__ in (
"pyupgrade._plugins.exceptions",
):
FUNCS[tp].append(func)
return func
[1] https://github.com/asottile/pyupgrade
author | Matt Harbison <matt_harbison@yahoo.com> |
---|---|
date | Sun, 05 Jan 2025 21:03:17 -0500 |
parents | 1a640aa20e48 |
children | 5cc8deb96b48 |
rev | line source |
---|---|
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
1 # This library is free software; you can redistribute it and/or |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
2 # modify it under the terms of the GNU Lesser General Public |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
3 # License as published by the Free Software Foundation; either |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
4 # version 2.1 of the License, or (at your option) any later version. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
5 # |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
6 # This library is distributed in the hope that it will be useful, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
9 # Lesser General Public License for more details. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
10 # |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
11 # You should have received a copy of the GNU Lesser General Public |
15782
7de7630053cb
Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents:
14958
diff
changeset
|
12 # License along with this library; if not, see |
7de7630053cb
Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents:
14958
diff
changeset
|
13 # <http://www.gnu.org/licenses/>. |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
14 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
15 # This file is part of urlgrabber, a high-level cross-protocol url-grabber |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
16 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
17 |
4026
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
18 # Modified by Benoit Boissinot: |
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
19 # - fix for digest auth (inspired from urllib2.py @ Python v2.4) |
6470
ac0bcd951c2c
python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
6001
diff
changeset
|
20 # Modified by Dirkjan Ochtman: |
ac0bcd951c2c
python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
6001
diff
changeset
|
21 # - import md5 function from a local util module |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
22 # Modified by Augie Fackler: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
23 # - add safesend method and use it to prevent broken pipe errors |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
24 # on large POST requests |
4026
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
25 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
26 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
27 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
28 >>> import urllib2 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
29 >>> from keepalive import HTTPHandler |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
30 >>> keepalive_handler = HTTPHandler() |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
31 >>> opener = urlreq.buildopener(keepalive_handler) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
32 >>> urlreq.installopener(opener) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
33 >>> |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
34 >>> fo = urlreq.urlopen('http://www.python.org') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
35 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
36 If a connection to a given host is requested, and all of the existing |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
37 connections are still in use, another connection will be opened. If |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
38 the handler tries to use an existing connection but it fails in some |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
39 way, it will be closed and removed from the pool. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
40 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
41 To remove the handler, simply re-run build_opener with no arguments, and |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
42 install that opener. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
43 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
44 You can explicitly close connections by using the close_connection() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
45 method of the returned file-like object (described below) or you can |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
46 use the handler methods: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
47 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
48 close_connection(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
49 close_all() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
50 open_connections() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
51 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
52 NOTE: using the close_connection and close_all methods of the handler |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
53 should be done with care when using multiple threads. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
54 * there is nothing that prevents another thread from creating new |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
55 connections immediately after connections are closed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
56 * no checks are done to prevent in-use connections from being closed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
57 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
58 >>> keepalive_handler.close_all() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
59 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
60 EXTRA ATTRIBUTES AND METHODS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
61 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
62 Upon a status of 200, the object returned has a few additional |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
63 attributes and methods, which should not be used if you want to |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
64 remain consistent with the normal urllib2-returned objects: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
65 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
66 close_connection() - close the connection to the host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
67 readlines() - you know, readlines() |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
68 status - the return status (i.e. 404) |
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
69 reason - english translation of status (i.e. 'File not found') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
70 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
71 If you want the best of both worlds, use this inside an |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
72 AttributeError-catching try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
73 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
74 >>> try: status = fo.status |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
75 >>> except AttributeError: status = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
76 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
77 Unfortunately, these are ONLY there if status == 200, so it's not |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
78 easy to distinguish between non-200 responses. The reason is that |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
79 urllib2 tries to do clever things with error codes 301, 302, 401, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
80 and 407, and it wraps the object upon return. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
81 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
82 |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
83 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $ |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
84 |
51859
f4733654f144
typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents:
51858
diff
changeset
|
85 from __future__ import annotations |
27507
a16489f9132d
keepalive: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
86 |
41408
a43acfa2b76d
keepalive: use collections.defaultdict for host map
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40874
diff
changeset
|
87 import collections |
29341
0d83ad967bf8
cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
Augie Fackler <raf@durin42.com>
parents:
28883
diff
changeset
|
88 import hashlib |
27507
a16489f9132d
keepalive: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
89 import sys |
29456
e61d384e3be4
keepalive: switch from thread to threading module
Pulkit Goyal <7895pulkit@gmail.com>
parents:
29455
diff
changeset
|
90 import threading |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
91 |
34308
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
92 from .i18n import _ |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
93 from .node import hex |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
94 from . import ( |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
95 pycompat, |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
96 urllibcompat, |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
97 util, |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
98 ) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
99 from .utils import procutil |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
100 |
29455
0c741fd6158a
py3: conditionalize httplib import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
29341
diff
changeset
|
101 httplib = util.httplib |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
102 urlerr = util.urlerr |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
103 urlreq = util.urlreq |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
104 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
105 DEBUG = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
106 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
107 |
48946
642e31cb55f0
py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents:
48924
diff
changeset
|
108 class ConnectionManager: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
109 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
110 The connection manager must be able to: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
111 * keep track of all existing |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45352
diff
changeset
|
112 """ |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
113 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
114 def __init__(self): |
29456
e61d384e3be4
keepalive: switch from thread to threading module
Pulkit Goyal <7895pulkit@gmail.com>
parents:
29455
diff
changeset
|
115 self._lock = threading.Lock() |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
116 self._hostmap = collections.defaultdict(list) # host -> [connection] |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
117 self._connmap = {} # map connections to host |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
118 self._readymap = {} # map connection to ready state |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
119 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
120 def add(self, host, connection, ready): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
121 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
122 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
123 self._hostmap[host].append(connection) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
124 self._connmap[connection] = host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
125 self._readymap[connection] = ready |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
126 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
127 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
128 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
129 def remove(self, connection): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
130 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
131 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
132 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
133 host = self._connmap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
134 except KeyError: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
135 pass |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
136 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
137 del self._connmap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
138 del self._readymap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
139 self._hostmap[host].remove(connection) |
34435
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
140 if not self._hostmap[host]: |
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
141 del self._hostmap[host] |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
142 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
143 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
144 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
145 def set_ready(self, connection, ready): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
146 try: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
147 self._readymap[connection] = ready |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
148 except KeyError: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
149 pass |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
150 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
151 def get_ready_conn(self, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
152 conn = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
153 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
154 try: |
41408
a43acfa2b76d
keepalive: use collections.defaultdict for host map
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40874
diff
changeset
|
155 for c in self._hostmap[host]: |
a43acfa2b76d
keepalive: use collections.defaultdict for host map
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40874
diff
changeset
|
156 if self._readymap[c]: |
41409
1db94ebbc207
keepalive: track ready state with a bool
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41408
diff
changeset
|
157 self._readymap[c] = False |
41408
a43acfa2b76d
keepalive: use collections.defaultdict for host map
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40874
diff
changeset
|
158 conn = c |
a43acfa2b76d
keepalive: use collections.defaultdict for host map
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40874
diff
changeset
|
159 break |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
160 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
161 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
162 return conn |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
163 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
164 def get_all(self, host=None): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
165 if host: |
41408
a43acfa2b76d
keepalive: use collections.defaultdict for host map
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40874
diff
changeset
|
166 return list(self._hostmap[host]) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
167 else: |
49542
8251f7cc787d
keepalive: ensure `close_all()` actually closes all cached connections
Matt Harbison <matt_harbison@yahoo.com>
parents:
49304
diff
changeset
|
168 return dict( |
8251f7cc787d
keepalive: ensure `close_all()` actually closes all cached connections
Matt Harbison <matt_harbison@yahoo.com>
parents:
49304
diff
changeset
|
169 {h: list(conns) for (h, conns) in self._hostmap.items()} |
8251f7cc787d
keepalive: ensure `close_all()` actually closes all cached connections
Matt Harbison <matt_harbison@yahoo.com>
parents:
49304
diff
changeset
|
170 ) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
171 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
172 |
48946
642e31cb55f0
py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents:
48924
diff
changeset
|
173 class KeepAliveHandler: |
40043
6509fcec830c
url: allow to configure timeout on http connection
C?dric Krier <ced@b2ck.com>
parents:
40033
diff
changeset
|
174 def __init__(self, timeout=None): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
175 self._cm = ConnectionManager() |
40043
6509fcec830c
url: allow to configure timeout on http connection
C?dric Krier <ced@b2ck.com>
parents:
40033
diff
changeset
|
176 self._timeout = timeout |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
177 self.requestscount = 0 |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
178 self.sentbytescount = 0 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
179 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
180 #### Connection Management |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
181 def open_connections(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
182 """return a list of connected hosts and the number of connections |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
183 to each. [('foo.com:80', 2), ('bar.org', 1)]""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
184 return [(host, len(li)) for (host, li) in self._cm.get_all().items()] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
185 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
186 def close_connection(self, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
187 """close connection(s) to <host> |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
188 host is the host:port spec, as in 'www.cnn.com:8080' as passed in. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
189 no error occurs if there is no connection to that host.""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
190 for h in self._cm.get_all(host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
191 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
192 h.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
193 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
194 def close_all(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
195 """close all open connections""" |
48924
dea766fca7e1
keepalive: remove pycompat.iteritems()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
48895
diff
changeset
|
196 for host, conns in self._cm.get_all().items(): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
197 for h in conns: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
198 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
199 h.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
200 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
201 def _request_closed(self, request, host, connection): |
17251
98166640b356
help: fix some instances of 'the the'
Mads Kiilerich <mads@kiilerich.com>
parents:
16705
diff
changeset
|
202 """tells us that this request is now closed and that the |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
203 connection is ready for another request""" |
41409
1db94ebbc207
keepalive: track ready state with a bool
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41408
diff
changeset
|
204 self._cm.set_ready(connection, True) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
205 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
206 def _remove_connection(self, host, connection, close=0): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
207 if close: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
208 connection.close() |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
209 self._cm.remove(connection) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
210 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
211 #### Transaction Execution |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
212 def http_open(self, req): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
213 return self.do_open(HTTPConnection, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
214 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
215 def do_open(self, http_class, req): |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
216 host = urllibcompat.gethost(req) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
217 if not host: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
218 raise urlerr.urlerror(b'no host given') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
219 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
220 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
221 h = self._cm.get_ready_conn(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
222 while h: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
223 r = self._reuse_connection(h, req, host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
224 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
225 # if this response is non-None, then it worked and we're |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
226 # done. Break out, skipping the else block. |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
227 if r: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
228 break |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
229 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
230 # connection is bad - possibly closed by server |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
231 # discard it and ask for the next free connection |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
232 h.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
233 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
234 h = self._cm.get_ready_conn(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
235 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
236 # no (working) free connections were found. Create a new one. |
40043
6509fcec830c
url: allow to configure timeout on http connection
C?dric Krier <ced@b2ck.com>
parents:
40033
diff
changeset
|
237 h = http_class(host, timeout=self._timeout) |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
238 if DEBUG: |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
239 DEBUG.info( |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
240 b"creating new connection to %s (%d)", host, id(h) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
241 ) |
41409
1db94ebbc207
keepalive: track ready state with a bool
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41408
diff
changeset
|
242 self._cm.add(host, h, False) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
243 self._start_transaction(h, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
244 r = h.getresponse() |
34308
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
245 # The string form of BadStatusLine is the status line. Add some context |
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
246 # to make the error message slightly more useful. |
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
247 except httplib.BadStatusLine as err: |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
248 raise urlerr.urlerror( |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
249 _(b'bad HTTP status line: %s') % pycompat.sysbytes(err.line) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
250 ) |
52640
24ee91ba9aa8
pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents:
51875
diff
changeset
|
251 except (OSError, httplib.HTTPException) as err: |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
252 raise urlerr.urlerror(err) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
253 |
39649
d6d094259d9c
keepalive: work around slight deficiency in vcr
Augie Fackler <raf@durin42.com>
parents:
37688
diff
changeset
|
254 # If not a persistent connection, don't try to reuse it. Look |
d6d094259d9c
keepalive: work around slight deficiency in vcr
Augie Fackler <raf@durin42.com>
parents:
37688
diff
changeset
|
255 # for this using getattr() since vcr doesn't define this |
d6d094259d9c
keepalive: work around slight deficiency in vcr
Augie Fackler <raf@durin42.com>
parents:
37688
diff
changeset
|
256 # attribute, and in that case always close the connection. |
43103
c95b2f40db7c
py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents:
43089
diff
changeset
|
257 if getattr(r, 'will_close', True): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
258 self._cm.remove(h) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
259 |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
260 if DEBUG: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
261 DEBUG.info(b"STATUS: %s, %s", r.status, r.reason) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
262 r._handler = self |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
263 r._host = host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
264 r._url = req.get_full_url() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
265 r._connection = h |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
266 r.code = r.status |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
267 r.headers = r.msg |
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
268 r.msg = r.reason |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
269 |
30487
88a448a12ae8
keepalive: discard legacy Python support for error handling
Augie Fackler <augie@google.com>
parents:
30473
diff
changeset
|
270 return r |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
271 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
272 def _reuse_connection(self, h, req, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
273 """start the transaction with a re-used connection |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
274 return a response object (r) upon success or None on failure. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
275 This DOES not close or remove bad connections in cases where |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
276 it returns. However, if an unexpected exception occurs, it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
277 will close and remove the connection before re-raising. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
278 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
279 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
280 self._start_transaction(h, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
281 r = h.getresponse() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
282 # note: just because we got something back doesn't mean it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
283 # worked. We'll check the version below, too. |
52640
24ee91ba9aa8
pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents:
51875
diff
changeset
|
284 except (OSError, httplib.HTTPException): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
285 r = None |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
286 except: # re-raises |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
287 # adding this block just in case we've missed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
288 # something we will still raise the exception, but |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
289 # lets try and close the connection and remove it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
290 # first. We previously got into a nasty loop |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
291 # where an exception was uncaught, and so the |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
292 # connection stayed open. On the next try, the |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
293 # same exception was raised, etc. The trade-off is |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
294 # that it's now possible this call will raise |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
295 # a DIFFERENT exception |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
296 if DEBUG: |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
297 DEBUG.error( |
43117
8ff1ecfadcd1
cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents:
43106
diff
changeset
|
298 b"unexpected exception - closing connection to %s (%d)", |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
299 host, |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
300 id(h), |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
301 ) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
302 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
303 h.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
304 raise |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
305 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
306 if r is None or r.version == 9: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
307 # httplib falls back to assuming HTTP 0.9 if it gets a |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
308 # bad header back. This is most likely to happen if |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
309 # the socket has been closed by the server since we |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
310 # last used the connection. |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
311 if DEBUG: |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
312 DEBUG.info( |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
313 b"failed to re-use connection to %s (%d)", host, id(h) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
314 ) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
315 r = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
316 else: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
317 if DEBUG: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
318 DEBUG.info(b"re-using connection to %s (%d)", host, id(h)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
319 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
320 return r |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
321 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
322 def _start_transaction(self, h, req): |
40379
41506e3b04ee
keepalive: use getattr to avoid AttributeErrors when vcr is in use
Augie Fackler <augie@google.com>
parents:
40043
diff
changeset
|
323 oldbytescount = getattr(h, 'sentbytescount', 0) |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
324 |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
325 # What follows mostly reimplements HTTPConnection.request() |
31999
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
326 # except it adds self.parent.addheaders in the mix and sends headers |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
327 # in a deterministic order (to make testing easier). |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
328 headers = util.sortdict(self.parent.addheaders) |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
329 headers.update(sorted(req.headers.items())) |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
330 headers.update(sorted(req.unredirected_hdrs.items())) |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
331 headers = util.sortdict((n.lower(), v) for n, v in headers.items()) |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
332 skipheaders = {} |
43506
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43117
diff
changeset
|
333 for n in ('host', 'accept-encoding'): |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
334 if n in headers: |
43506
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43117
diff
changeset
|
335 skipheaders['skip_' + n.replace('-', '_')] = 1 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
336 try: |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
337 if urllibcompat.hasdata(req): |
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
338 data = urllibcompat.getdata(req) |
30922
1beeb5185930
keepalive: honor urllib2 style get_method overrides
John Mulligan <phlogistonjohn@asynchrono.us>
parents:
30688
diff
changeset
|
339 h.putrequest( |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
340 req.get_method(), |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
341 urllibcompat.getselector(req), |
51858
607e94e01851
format: add many "missing" comma
Matt Harbison <matt_harbison@yahoo.com>
parents:
51830
diff
changeset
|
342 **skipheaders, |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
343 ) |
43506
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43117
diff
changeset
|
344 if 'content-type' not in headers: |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
345 h.putheader( |
43506
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43117
diff
changeset
|
346 'Content-type', 'application/x-www-form-urlencoded' |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
347 ) |
43506
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43117
diff
changeset
|
348 if 'content-length' not in headers: |
9f70512ae2cf
cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents:
43117
diff
changeset
|
349 h.putheader('Content-length', '%d' % len(data)) |
8146
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
350 else: |
30922
1beeb5185930
keepalive: honor urllib2 style get_method overrides
John Mulligan <phlogistonjohn@asynchrono.us>
parents:
30688
diff
changeset
|
351 h.putrequest( |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
352 req.get_method(), |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
353 urllibcompat.getselector(req), |
51858
607e94e01851
format: add many "missing" comma
Matt Harbison <matt_harbison@yahoo.com>
parents:
51830
diff
changeset
|
354 **skipheaders, |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
355 ) |
52640
24ee91ba9aa8
pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents:
51875
diff
changeset
|
356 except OSError as err: |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
357 raise urlerr.urlerror(err) |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
358 for k, v in headers.items(): |
8146
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
359 h.putheader(k, v) |
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
360 h.endheaders() |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
361 if urllibcompat.hasdata(req): |
8146
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
362 h.send(data) |
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
363 |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
364 # This will fail to record events in case of I/O failure. That's OK. |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
365 self.requestscount += 1 |
40379
41506e3b04ee
keepalive: use getattr to avoid AttributeErrors when vcr is in use
Augie Fackler <augie@google.com>
parents:
40043
diff
changeset
|
366 self.sentbytescount += getattr(h, 'sentbytescount', 0) - oldbytescount |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
367 |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
368 try: |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
369 self.parent.requestscount += 1 |
40379
41506e3b04ee
keepalive: use getattr to avoid AttributeErrors when vcr is in use
Augie Fackler <augie@google.com>
parents:
40043
diff
changeset
|
370 self.parent.sentbytescount += ( |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
371 getattr(h, 'sentbytescount', 0) - oldbytescount |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
372 ) |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
373 except AttributeError: |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
374 pass |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
375 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
376 |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
377 class HTTPHandler(KeepAliveHandler, urlreq.httphandler): |
5983
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
378 pass |
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
379 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
380 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
381 class HTTPResponse(httplib.HTTPResponse): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
382 # we need to subclass HTTPResponse in order to |
51827 | 383 # 1) add close_connection() method |
384 # 2) add geturl() method | |
51825
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
385 # 3) add accounting for read(), readlines() and readinto() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
386 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
387 def __init__(self, sock, debuglevel=0, strict=0, method=None): |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
388 httplib.HTTPResponse.__init__( |
48895
62baa61efe8f
keepalive: remove Python 2 support code
Gregory Szorc <gregory.szorc@gmail.com>
parents:
48875
diff
changeset
|
389 self, sock, debuglevel=debuglevel, method=method |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
390 ) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
391 self.fileno = sock.fileno |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
392 self.code = None |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
393 self.receivedbytescount = 0 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
394 self._rbuf = b'' |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
395 self._rbufsize = 8096 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
396 self._handler = None # inserted by the handler later |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
397 self._host = None # (same) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
398 self._url = None # (same) |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
399 self._connection = None # (same) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
400 |
41441
44d752efdbce
keepalive: implement _close_conn() so closes are known
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41409
diff
changeset
|
401 # Python 2.7 has a single close() which closes the socket handle. |
44d752efdbce
keepalive: implement _close_conn() so closes are known
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41409
diff
changeset
|
402 # This method was effectively renamed to _close_conn() in Python 3. But |
44d752efdbce
keepalive: implement _close_conn() so closes are known
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41409
diff
changeset
|
403 # there is also a close(). _close_conn() is called by methods like |
44d752efdbce
keepalive: implement _close_conn() so closes are known
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41409
diff
changeset
|
404 # read(). |
44d752efdbce
keepalive: implement _close_conn() so closes are known
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41409
diff
changeset
|
405 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
406 def close(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
407 if self.fp: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
408 self.fp.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
409 self.fp = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
410 if self._handler: |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
411 self._handler._request_closed( |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
412 self, self._host, self._connection |
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
413 ) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
414 |
41441
44d752efdbce
keepalive: implement _close_conn() so closes are known
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41409
diff
changeset
|
415 def _close_conn(self): |
44d752efdbce
keepalive: implement _close_conn() so closes are known
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41409
diff
changeset
|
416 self.close() |
44d752efdbce
keepalive: implement _close_conn() so closes are known
Gregory Szorc <gregory.szorc@gmail.com>
parents:
41409
diff
changeset
|
417 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
418 def close_connection(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
419 self._handler._remove_connection(self._host, self._connection, close=1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
420 self.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
421 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
422 def geturl(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
423 return self._url |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
424 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
425 def read(self, amt=None): |
51825
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
426 data = super().read(amt) |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
427 self.receivedbytescount += len(data) |
51827 | 428 if self._connection is not None: |
40426
588f1e9a4d16
http: work around custom http client classes that refuse extra attrs
Augie Fackler <augie@google.com>
parents:
40379
diff
changeset
|
429 self._connection.receivedbytescount += len(data) |
51827 | 430 if self._handler is not None: |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
431 self._handler.parent.receivedbytescount += len(data) |
51825
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
432 return data |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
433 |
51875
1a640aa20e48
keepalive: fix a signature mismatch for a http.client.HTTPResponse subclass
Matt Harbison <matt_harbison@yahoo.com>
parents:
51859
diff
changeset
|
434 def readline(self, limit: int = -1): |
1a640aa20e48
keepalive: fix a signature mismatch for a http.client.HTTPResponse subclass
Matt Harbison <matt_harbison@yahoo.com>
parents:
51859
diff
changeset
|
435 data = super().readline(limit=limit) |
51825
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
436 self.receivedbytescount += len(data) |
51827 | 437 if self._connection is not None: |
51825
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
438 self._connection.receivedbytescount += len(data) |
51827 | 439 if self._handler is not None: |
51825
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
440 self._handler.parent.receivedbytescount += len(data) |
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
441 return data |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
442 |
37297
97eedbd5a56c
keepalive: implement readinto()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37119
diff
changeset
|
443 def readinto(self, dest): |
51825
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
444 got = super().readinto(dest) |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
445 self.receivedbytescount += got |
51827 | 446 if self._connection is not None: |
51825
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
447 self._connection.receivedbytescount += got |
51827 | 448 if self._handler is not None: |
51825
c1ed5ee2ad82
http: reuse Python's implementation of read/readline/readinto
Joerg Sonnenberger <joerg@bec.de>
parents:
51823
diff
changeset
|
449 self._handler.parent.receivedbytescount += got |
37587
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
450 return got |
37297
97eedbd5a56c
keepalive: implement readinto()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37119
diff
changeset
|
451 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
452 |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
453 def safesend(self, str): |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
454 """Send `str' to the server. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
455 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
456 Shamelessly ripped off from httplib to patch a bad behavior. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
457 """ |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
458 # _broken_pipe_resp is an attribute we set in this function |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
459 # if the socket is closed while we're sending data but |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
460 # the server sent us a response before hanging up. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
461 # In that case, we want to pretend to send the rest of the |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
462 # outgoing data, and then let the user use getresponse() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
463 # (which we wrap) to get this last response before |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
464 # opening a new socket. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
465 if getattr(self, '_broken_pipe_resp', None) is not None: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
466 return |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
467 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
468 if self.sock is None: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
469 if self.auto_open: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
470 self.connect() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
471 else: |
16687
e34106fa0dc3
cleanup: "raise SomeException()" -> "raise SomeException"
Brodie Rao <brodie@sf.io>
parents:
16686
diff
changeset
|
472 raise httplib.NotConnected |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
473 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
474 # send the data to the server. if we get a broken pipe, then close |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
475 # the socket. we want to reconnect when somebody tries to send again. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
476 # |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
477 # NOTE: we DO propagate the error, though, because we cannot simply |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
478 # ignore the error... the caller will know if they can retry. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
479 if self.debuglevel > 0: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
480 print(b"send:", repr(str)) |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
481 try: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
482 blocksize = 8192 |
14958
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
483 read = getattr(str, 'read', None) |
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
484 if read is not None: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
485 if self.debuglevel > 0: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
486 print(b"sending a read()able") |
14958
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
487 data = read(blocksize) |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
488 while data: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
489 self.sock.sendall(data) |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
490 self.sentbytescount += len(data) |
14958
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
491 data = read(blocksize) |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
492 else: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
493 self.sock.sendall(str) |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
494 self.sentbytescount += len(str) |
49304
48f1b314056b
py3: catch BrokenPipeError instead of checking errno == EPIPE
Manuel Jacob <me@manueljacob.de>
parents:
48946
diff
changeset
|
495 except BrokenPipeError: |
48f1b314056b
py3: catch BrokenPipeError instead of checking errno == EPIPE
Manuel Jacob <me@manueljacob.de>
parents:
48946
diff
changeset
|
496 if self._HTTPConnection__state == httplib._CS_REQ_SENT: |
48f1b314056b
py3: catch BrokenPipeError instead of checking errno == EPIPE
Manuel Jacob <me@manueljacob.de>
parents:
48946
diff
changeset
|
497 self._broken_pipe_resp = None |
48f1b314056b
py3: catch BrokenPipeError instead of checking errno == EPIPE
Manuel Jacob <me@manueljacob.de>
parents:
48946
diff
changeset
|
498 self._broken_pipe_resp = self.getresponse() |
48f1b314056b
py3: catch BrokenPipeError instead of checking errno == EPIPE
Manuel Jacob <me@manueljacob.de>
parents:
48946
diff
changeset
|
499 reraise = False |
48f1b314056b
py3: catch BrokenPipeError instead of checking errno == EPIPE
Manuel Jacob <me@manueljacob.de>
parents:
48946
diff
changeset
|
500 else: |
48f1b314056b
py3: catch BrokenPipeError instead of checking errno == EPIPE
Manuel Jacob <me@manueljacob.de>
parents:
48946
diff
changeset
|
501 reraise = True |
48f1b314056b
py3: catch BrokenPipeError instead of checking errno == EPIPE
Manuel Jacob <me@manueljacob.de>
parents:
48946
diff
changeset
|
502 self.close() |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
503 if reraise: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
504 raise |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
505 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
506 |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
507 def wrapgetresponse(cls): |
45942
89a2afe31e82
formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents:
45352
diff
changeset
|
508 """Wraps getresponse in cls with a broken-pipe sane version.""" |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
509 |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
510 def safegetresponse(self): |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
511 # In safesend() we might set the _broken_pipe_resp |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
512 # attribute, in which case the socket has already |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
513 # been closed and we just need to give them the response |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
514 # back. Otherwise, we use the normal response path. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
515 r = getattr(self, '_broken_pipe_resp', None) |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
516 if r is not None: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
517 return r |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
518 return cls.getresponse(self) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
519 |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
520 safegetresponse.__doc__ = cls.getresponse.__doc__ |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
521 return safegetresponse |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
522 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
523 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
524 class HTTPConnection(httplib.HTTPConnection): |
40031
f2dffa1359c6
url: have httpsconnection inherit from our custom HTTPConnection
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39816
diff
changeset
|
525 # url.httpsconnection inherits from this. So when adding/removing |
f2dffa1359c6
url: have httpsconnection inherit from our custom HTTPConnection
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39816
diff
changeset
|
526 # attributes, be sure to audit httpsconnection() for unintended |
f2dffa1359c6
url: have httpsconnection inherit from our custom HTTPConnection
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39816
diff
changeset
|
527 # consequences. |
f2dffa1359c6
url: have httpsconnection inherit from our custom HTTPConnection
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39816
diff
changeset
|
528 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
529 # use the modified response class |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
530 response_class = HTTPResponse |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
531 send = safesend |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
532 getresponse = wrapgetresponse(httplib.HTTPConnection) |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
533 |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
534 def __init__(self, *args, **kwargs): |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
535 httplib.HTTPConnection.__init__(self, *args, **kwargs) |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
536 self.sentbytescount = 0 |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
537 self.receivedbytescount = 0 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
538 |
49543
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
539 def __repr__(self): |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
540 base = super(HTTPConnection, self).__repr__() |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
541 local = "(unconnected)" |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
542 s = self.sock |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
543 if s: |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
544 try: |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
545 local = "%s:%d" % s.getsockname() |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
546 except OSError: |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
547 pass # Likely not connected |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
548 return "<%s: %s <--> %s:%d>" % (base, local, self.host, self.port) |
76fbb1b6692a
keepalive: add `__repr__()` to the HTTPConnection class to ease debugging
Matt Harbison <matt_harbison@yahoo.com>
parents:
49542
diff
changeset
|
549 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
550 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
551 ######################################################################### |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
552 ##### TEST FUNCTIONS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
553 ######################################################################### |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
554 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
555 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
556 def continuity(url): |
29341
0d83ad967bf8
cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
Augie Fackler <raf@durin42.com>
parents:
28883
diff
changeset
|
557 md5 = hashlib.md5 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
558 format = b'%25s: %s' |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
559 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
560 # first fetch the file with the normal http handler |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
561 opener = urlreq.buildopener() |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
562 urlreq.installopener(opener) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
563 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
564 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
565 fo.close() |
22505
232d437af120
keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents:
19872
diff
changeset
|
566 m = md5(foo) |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
567 print(format % (b'normal urllib', hex(m.digest()))) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
568 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
569 # now install the keepalive handler and try again |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
570 opener = urlreq.buildopener(HTTPHandler()) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
571 urlreq.installopener(opener) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
572 |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
573 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
574 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
575 fo.close() |
22505
232d437af120
keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents:
19872
diff
changeset
|
576 m = md5(foo) |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
577 print(format % (b'keepalive read', hex(m.digest()))) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
578 |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
579 fo = urlreq.urlopen(url) |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
580 foo = b'' |
14494
1ffeeb91c55d
check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents:
10394
diff
changeset
|
581 while True: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
582 f = fo.readline() |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
583 if f: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
584 foo = foo + f |
34435
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
585 else: |
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
586 break |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
587 fo.close() |
22505
232d437af120
keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents:
19872
diff
changeset
|
588 m = md5(foo) |
46113
59fa3890d40a
node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents:
45942
diff
changeset
|
589 print(format % (b'keepalive readline', hex(m.digest()))) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
590 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
591 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
592 def comp(N, url): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
593 print(b' making %i connections to:\n %s' % (N, url)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
594 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
595 procutil.stdout.write(b' first using the normal urllib handlers') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
596 # first use normal opener |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
597 opener = urlreq.buildopener() |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
598 urlreq.installopener(opener) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
599 t1 = fetch(N, url) |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
600 print(b' TIME: %.3f s' % t1) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
601 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
602 procutil.stdout.write(b' now using the keepalive handler ') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
603 # now install the keepalive handler and try again |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
604 opener = urlreq.buildopener(HTTPHandler()) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
605 urlreq.installopener(opener) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
606 t2 = fetch(N, url) |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
607 print(b' TIME: %.3f s' % t2) |
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
608 print(b' improvement factor: %.2f' % (t1 / t2)) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
609 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
610 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
611 def fetch(N, url, delay=0): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
612 import time |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
613 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
614 lens = [] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
615 starttime = time.time() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
616 for i in range(N): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
617 if delay and i > 0: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
618 time.sleep(delay) |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
619 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
620 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
621 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
622 lens.append(len(foo)) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
623 diff = time.time() - starttime |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
624 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
625 j = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
626 for i in lens[1:]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
627 j = j + 1 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
628 if not i == lens[0]: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
629 print(b"WARNING: inconsistent length on read %i: %i" % (j, i)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
630 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
631 return diff |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
632 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
633 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
634 def test_timeout(url): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
635 global DEBUG |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
636 dbbackup = DEBUG |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
637 |
48946
642e31cb55f0
py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents:
48924
diff
changeset
|
638 class FakeLogger: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
639 def debug(self, msg, *args): |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
640 print(msg % args) |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
641 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
642 info = warning = error = debug |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
643 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
644 DEBUG = FakeLogger() |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
645 print(b" fetching the file to establish a connection") |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
646 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
647 data1 = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
648 fo.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
649 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
650 i = 20 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
651 print(b" waiting %i seconds for the server to close the connection" % i) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
652 while i > 0: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
653 procutil.stdout.write(b'\r %2i' % i) |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
654 procutil.stdout.flush() |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
655 time.sleep(1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
656 i -= 1 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
657 procutil.stderr.write(b'\r') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
658 |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
659 print(b" fetching the file a second time") |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
660 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
661 data2 = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
662 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
663 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
664 if data1 == data2: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
665 print(b' data are identical') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
666 else: |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
667 print(b' ERROR: DATA DIFFER') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
668 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
669 DEBUG = dbbackup |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
670 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
671 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
672 def test(url, N=10): |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
673 print(b"performing continuity test (making sure stuff isn't corrupted)") |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
674 continuity(url) |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
675 print(b'') |
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
676 print(b"performing speed comparison") |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
677 comp(N, url) |
43077
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
678 print(b'') |
687b865b95ad
formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents:
43076
diff
changeset
|
679 print(b"performing dropped-connection check") |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
680 test_timeout(url) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
681 |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
682 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
683 if __name__ == '__main__': |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
684 import time |
43076
2372284d9457
formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents:
41441
diff
changeset
|
685 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
686 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
687 N = int(sys.argv[1]) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
688 url = sys.argv[2] |
16688
cfb6682961b8
cleanup: replace naked excepts with more specific ones
Brodie Rao <brodie@sf.io>
parents:
16687
diff
changeset
|
689 except (IndexError, ValueError): |
51790
bd490cdf764a
py3: use str literal instead of bytes literal
Manuel Jacob <me@manueljacob.de>
parents:
50926
diff
changeset
|
690 print("%s <integer> <url>" % sys.argv[0]) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
691 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
692 test(url, N) |