mercurial-scm/hg-stable: mercurial/keepalive.py annotate

annotate mercurial/keepalive.py @ 51875:454feddab720

brancing: merge stable into default

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Fri, 06 Sep 2024 02:12:19 +0200
parents	34eb3a711955 bd490cdf764a
children	607e94e01851

rev	line source
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	1 # This library is free software; you can redistribute it and/or
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	2 # modify it under the terms of the GNU Lesser General Public
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	3 # License as published by the Free Software Foundation; either
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	4 # version 2.1 of the License, or (at your option) any later version.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	5 #
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	6 # This library is distributed in the hope that it will be useful,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	7 # but WITHOUT ANY WARRANTY; without even the implied warranty of
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	9 # Lesser General Public License for more details.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	10 #
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	11 # You should have received a copy of the GNU Lesser General Public
15782 7de7630053cb Remove FSF mailing address from GPL headers Martin Geisler <mg@aragost.com> parents: 14958 diff changeset	12 # License along with this library; if not, see
7de7630053cb Remove FSF mailing address from GPL headers Martin Geisler <mg@aragost.com> parents: 14958 diff changeset	13 # <http://www.gnu.org/licenses/>.
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	14
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	15 # This file is part of urlgrabber, a high-level cross-protocol url-grabber
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	16 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	17
4026 8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	18 # Modified by Benoit Boissinot:
8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	19 # - fix for digest auth (inspired from urllib2.py @ Python v2.4)
6470 ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions Dirkjan Ochtman <dirkjan@ochtman.nl> parents: 6001 diff changeset	20 # Modified by Dirkjan Ochtman:
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions Dirkjan Ochtman <dirkjan@ochtman.nl> parents: 6001 diff changeset	21 # - import md5 function from a local util module
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	22 # Modified by Augie Fackler:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	23 # - add safesend method and use it to prevent broken pipe errors
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	24 # on large POST requests
4026 8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	25
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	26 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	27
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	28 >>> import urllib2
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	29 >>> from keepalive import HTTPHandler
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	30 >>> keepalive_handler = HTTPHandler()
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	31 >>> opener = urlreq.buildopener(keepalive_handler)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	32 >>> urlreq.installopener(opener)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	33 >>>
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	34 >>> fo = urlreq.urlopen('http://www.python.org')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	35
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	36 If a connection to a given host is requested, and all of the existing
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	37 connections are still in use, another connection will be opened. If
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	38 the handler tries to use an existing connection but it fails in some
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	39 way, it will be closed and removed from the pool.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	40
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	41 To remove the handler, simply re-run build_opener with no arguments, and
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	42 install that opener.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	43
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	44 You can explicitly close connections by using the close_connection()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	45 method of the returned file-like object (described below) or you can
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	46 use the handler methods:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	47
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	48 close_connection(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	49 close_all()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	50 open_connections()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	51
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	52 NOTE: using the close_connection and close_all methods of the handler
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	53 should be done with care when using multiple threads.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	54 * there is nothing that prevents another thread from creating new
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	55 connections immediately after connections are closed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	56 * no checks are done to prevent in-use connections from being closed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	57
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	58 >>> keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	59
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	60 EXTRA ATTRIBUTES AND METHODS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	61
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	62 Upon a status of 200, the object returned has a few additional
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	63 attributes and methods, which should not be used if you want to
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	64 remain consistent with the normal urllib2-returned objects:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	65
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	66 close_connection() - close the connection to the host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	67 readlines() - you know, readlines()
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	68 status - the return status (i.e. 404)
e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	69 reason - english translation of status (i.e. 'File not found')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	70
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	71 If you want the best of both worlds, use this inside an
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	72 AttributeError-catching try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	73
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	74 >>> try: status = fo.status
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	75 >>> except AttributeError: status = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	76
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	77 Unfortunately, these are ONLY there if status == 200, so it's not
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	78 easy to distinguish between non-200 responses. The reason is that
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	79 urllib2 tries to do clever things with error codes 301, 302, 401,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	80 and 407, and it wraps the object upon return.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	81 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	82
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	83 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	84
27507 a16489f9132d keepalive: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	85
41430 a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40892 diff changeset	86 import collections
29341 0d83ad967bf8 cleanup: replace uses of util.(md5\|sha1\|sha256\|sha512) with hashlib.\1 Augie Fackler <raf@durin42.com> parents: 28883 diff changeset	87 import hashlib
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	88 import socket
27507 a16489f9132d keepalive: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	89 import sys
29456 e61d384e3be4 keepalive: switch from thread to threading module Pulkit Goyal <7895pulkit@gmail.com> parents: 29455 diff changeset	90 import threading
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	91
34315 9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32688 diff changeset	92 from .i18n import _
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 45957 diff changeset	93 from .node import hex
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	94 from . import (
34428 a454123f5d94 keepalive: python 3 portability tweaks Augie Fackler <augie@google.com> parents: 34338 diff changeset	95 pycompat,
34467 1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34436 diff changeset	96 urllibcompat,
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	97 util,
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	98 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	99 from .utils import procutil
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	100
29455 0c741fd6158a py3: conditionalize httplib import Pulkit Goyal <7895pulkit@gmail.com> parents: 29341 diff changeset	101 httplib = util.httplib
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	102 urlerr = util.urlerr
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	103 urlreq = util.urlreq
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	104
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	105 DEBUG = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	106
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	107
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 49015 diff changeset	108 class ConnectionManager:
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	109 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	110 The connection manager must be able to:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	111 * keep track of all existing
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45207 diff changeset	112 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	113
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	114 def __init__(self):
29456 e61d384e3be4 keepalive: switch from thread to threading module Pulkit Goyal <7895pulkit@gmail.com> parents: 29455 diff changeset	115 self._lock = threading.Lock()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	116 self._hostmap = collections.defaultdict(list) # host -> [connection]
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	117 self._connmap = {} # map connections to host
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	118 self._readymap = {} # map connection to ready state
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	119
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	120 def add(self, host, connection, ready):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	121 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	122 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	123 self._hostmap[host].append(connection)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	124 self._connmap[connection] = host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	125 self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	126 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	127 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	128
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	129 def remove(self, connection):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	130 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	131 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	132 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	133 host = self._connmap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	134 except KeyError:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	135 pass
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	136 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	137 del self._connmap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	138 del self._readymap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	139 self._hostmap[host].remove(connection)
34436 5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34428 diff changeset	140 if not self._hostmap[host]:
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34428 diff changeset	141 del self._hostmap[host]
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	142 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	143 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	144
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	145 def set_ready(self, connection, ready):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	146 try:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	147 self._readymap[connection] = ready
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	148 except KeyError:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	149 pass
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	150
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	151 def get_ready_conn(self, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	152 conn = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	153 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	154 try:
41430 a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40892 diff changeset	155 for c in self._hostmap[host]:
a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40892 diff changeset	156 if self._readymap[c]:
41431 1db94ebbc207 keepalive: track ready state with a bool Gregory Szorc <gregory.szorc@gmail.com> parents: 41430 diff changeset	157 self._readymap[c] = False
41430 a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40892 diff changeset	158 conn = c
a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40892 diff changeset	159 break
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	160 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	161 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	162 return conn
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	163
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	164 def get_all(self, host=None):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	165 if host:
41430 a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40892 diff changeset	166 return list(self._hostmap[host])
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	167 else:
49534 8251f7cc787d keepalive: ensure `close_all()` actually closes all cached connections Matt Harbison <matt_harbison@yahoo.com> parents: 49312 diff changeset	168 return dict(
8251f7cc787d keepalive: ensure `close_all()` actually closes all cached connections Matt Harbison <matt_harbison@yahoo.com> parents: 49312 diff changeset	169 {h: list(conns) for (h, conns) in self._hostmap.items()}
8251f7cc787d keepalive: ensure `close_all()` actually closes all cached connections Matt Harbison <matt_harbison@yahoo.com> parents: 49312 diff changeset	170 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	171
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	172
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 49015 diff changeset	173 class KeepAliveHandler:
40044 6509fcec830c url: allow to configure timeout on http connection C?dric Krier <ced@b2ck.com> parents: 40034 diff changeset	174 def __init__(self, timeout=None):
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	175 self._cm = ConnectionManager()
40044 6509fcec830c url: allow to configure timeout on http connection C?dric Krier <ced@b2ck.com> parents: 40034 diff changeset	176 self._timeout = timeout
40033 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	177 self.requestscount = 0
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	178 self.sentbytescount = 0
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	179
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	180 #### Connection Management
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	181 def open_connections(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	182 """return a list of connected hosts and the number of connections
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	183 to each. [('foo.com:80', 2), ('bar.org', 1)]"""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	184 return [(host, len(li)) for (host, li) in self._cm.get_all().items()]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	185
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	186 def close_connection(self, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	187 """close connection(s) to <host>
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	188 host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	189 no error occurs if there is no connection to that host."""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	190 for h in self._cm.get_all(host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	191 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	192 h.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	193
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	194 def close_all(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	195 """close all open connections"""
49015 dea766fca7e1 keepalive: remove pycompat.iteritems() Gregory Szorc <gregory.szorc@gmail.com> parents: 48986 diff changeset	196 for host, conns in self._cm.get_all().items():
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	197 for h in conns:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	198 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	199 h.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	200
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	201 def _request_closed(self, request, host, connection):
17251 98166640b356 help: fix some instances of 'the the' Mads Kiilerich <mads@kiilerich.com> parents: 16705 diff changeset	202 """tells us that this request is now closed and that the
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	203 connection is ready for another request"""
41431 1db94ebbc207 keepalive: track ready state with a bool Gregory Szorc <gregory.szorc@gmail.com> parents: 41430 diff changeset	204 self._cm.set_ready(connection, True)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	205
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	206 def _remove_connection(self, host, connection, close=0):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	207 if close:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	208 connection.close()
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	209 self._cm.remove(connection)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	210
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	211 #### Transaction Execution
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	212 def http_open(self, req):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	213 return self.do_open(HTTPConnection, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	214
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	215 def do_open(self, http_class, req):
34467 1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34436 diff changeset	216 host = urllibcompat.gethost(req)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	217 if not host:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	218 raise urlerr.urlerror(b'no host given')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	219
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	220 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	221 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	222 while h:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	223 r = self._reuse_connection(h, req, host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	224
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	225 # if this response is non-None, then it worked and we're
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	226 # done. Break out, skipping the else block.
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	227 if r:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	228 break
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	229
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	230 # connection is bad - possibly closed by server
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	231 # discard it and ask for the next free connection
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	232 h.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	233 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	234 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	235 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	236 # no (working) free connections were found. Create a new one.
40044 6509fcec830c url: allow to configure timeout on http connection C?dric Krier <ced@b2ck.com> parents: 40034 diff changeset	237 h = http_class(host, timeout=self._timeout)
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	238 if DEBUG:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	239 DEBUG.info(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	240 b"creating new connection to %s (%d)", host, id(h)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	241 )
41431 1db94ebbc207 keepalive: track ready state with a bool Gregory Szorc <gregory.szorc@gmail.com> parents: 41430 diff changeset	242 self._cm.add(host, h, False)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	243 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	244 r = h.getresponse()
34315 9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32688 diff changeset	245 # The string form of BadStatusLine is the status line. Add some context
9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32688 diff changeset	246 # to make the error message slightly more useful.
9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32688 diff changeset	247 except httplib.BadStatusLine as err:
34428 a454123f5d94 keepalive: python 3 portability tweaks Augie Fackler <augie@google.com> parents: 34338 diff changeset	248 raise urlerr.urlerror(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	249 _(b'bad HTTP status line: %s') % pycompat.sysbytes(err.line)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	250 )
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 22958 diff changeset	251 except (socket.error, httplib.HTTPException) as err:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	252 raise urlerr.urlerror(err)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	253
39665 d6d094259d9c keepalive: work around slight deficiency in vcr Augie Fackler <raf@durin42.com> parents: 37688 diff changeset	254 # If not a persistent connection, don't try to reuse it. Look
d6d094259d9c keepalive: work around slight deficiency in vcr Augie Fackler <raf@durin42.com> parents: 37688 diff changeset	255 # for this using getattr() since vcr doesn't define this
d6d094259d9c keepalive: work around slight deficiency in vcr Augie Fackler <raf@durin42.com> parents: 37688 diff changeset	256 # attribute, and in that case always close the connection.
43103 c95b2f40db7c py3: stop normalizing 2nd argument of attr() to unicode Gregory Szorc <gregory.szorc@gmail.com>* parents: 43089 diff changeset	257 if getattr(r, 'will_close', True):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	258 self._cm.remove(h)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	259
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	260 if DEBUG:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	261 DEBUG.info(b"STATUS: %s, %s", r.status, r.reason)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	262 r._handler = self
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	263 r._host = host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	264 r._url = req.get_full_url()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	265 r._connection = h
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	266 r.code = r.status
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	267 r.headers = r.msg
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	268 r.msg = r.reason
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	269
30496 88a448a12ae8 keepalive: discard legacy Python support for error handling Augie Fackler <augie@google.com> parents: 30482 diff changeset	270 return r
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	271
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	272 def _reuse_connection(self, h, req, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	273 """start the transaction with a re-used connection
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	274 return a response object (r) upon success or None on failure.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	275 This DOES not close or remove bad connections in cases where
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	276 it returns. However, if an unexpected exception occurs, it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	277 will close and remove the connection before re-raising.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	278 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	279 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	280 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	281 r = h.getresponse()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	282 # note: just because we got something back doesn't mean it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	283 # worked. We'll check the version below, too.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	284 except (socket.error, httplib.HTTPException):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	285 r = None
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	286 except: # re-raises
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	287 # adding this block just in case we've missed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	288 # something we will still raise the exception, but
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	289 # lets try and close the connection and remove it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	290 # first. We previously got into a nasty loop
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	291 # where an exception was uncaught, and so the
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	292 # connection stayed open. On the next try, the
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	293 # same exception was raised, etc. The trade-off is
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	294 # that it's now possible this call will raise
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	295 # a DIFFERENT exception
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	296 if DEBUG:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	297 DEBUG.error(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43106 diff changeset	298 b"unexpected exception - closing connection to %s (%d)",
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	299 host,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	300 id(h),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	301 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	302 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	303 h.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	304 raise
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	305
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	306 if r is None or r.version == 9:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	307 # httplib falls back to assuming HTTP 0.9 if it gets a
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	308 # bad header back. This is most likely to happen if
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	309 # the socket has been closed by the server since we
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	310 # last used the connection.
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	311 if DEBUG:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	312 DEBUG.info(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	313 b"failed to re-use connection to %s (%d)", host, id(h)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	314 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	315 r = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	316 else:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	317 if DEBUG:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	318 DEBUG.info(b"re-using connection to %s (%d)", host, id(h))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	319
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	320 return r
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	321
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	322 def _start_transaction(self, h, req):
40379 41506e3b04ee keepalive: use getattr to avoid AttributeErrors when vcr is in use Augie Fackler <augie@google.com> parents: 40044 diff changeset	323 oldbytescount = getattr(h, 'sentbytescount', 0)
40033 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	324
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	325 # What follows mostly reimplements HTTPConnection.request()
31999 aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	326 # except it adds self.parent.addheaders in the mix and sends headers
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	327 # in a deterministic order (to make testing easier).
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	328 headers = util.sortdict(self.parent.addheaders)
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	329 headers.update(sorted(req.headers.items()))
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	330 headers.update(sorted(req.unredirected_hdrs.items()))
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	331 headers = util.sortdict((n.lower(), v) for n, v in headers.items())
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	332 skipheaders = {}
43554 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	333 for n in ('host', 'accept-encoding'):
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	334 if n in headers:
43554 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	335 skipheaders['skip_' + n.replace('-', '_')] = 1
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	336 try:
34467 1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34436 diff changeset	337 if urllibcompat.hasdata(req):
1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34436 diff changeset	338 data = urllibcompat.getdata(req)
30942 1beeb5185930 keepalive: honor urllib2 style get_method overrides John Mulligan <phlogistonjohn@asynchrono.us> parents: 30688 diff changeset	339 h.putrequest(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	340 req.get_method(),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	341 urllibcompat.getselector(req),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	342 **skipheaders
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	343 )
43554 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	344 if 'content-type' not in headers:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	345 h.putheader(
43554 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	346 'Content-type', 'application/x-www-form-urlencoded'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	347 )
43554 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	348 if 'content-length' not in headers:
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	349 h.putheader('Content-length', '%d' % len(data))
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	350 else:
30942 1beeb5185930 keepalive: honor urllib2 style get_method overrides John Mulligan <phlogistonjohn@asynchrono.us> parents: 30688 diff changeset	351 h.putrequest(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	352 req.get_method(),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	353 urllibcompat.getselector(req),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	354 **skipheaders
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	355 )
28278 b1b22185c764 keepalive: remove useless parentheses around exception type Gregory Szorc <gregory.szorc@gmail.com> parents: 27616 diff changeset	356 except socket.error as err:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	357 raise urlerr.urlerror(err)
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	358 for k, v in headers.items():
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	359 h.putheader(k, v)
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	360 h.endheaders()
34467 1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34436 diff changeset	361 if urllibcompat.hasdata(req):
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	362 h.send(data)
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	363
40033 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	364 # This will fail to record events in case of I/O failure. That's OK.
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	365 self.requestscount += 1
40379 41506e3b04ee keepalive: use getattr to avoid AttributeErrors when vcr is in use Augie Fackler <augie@google.com> parents: 40044 diff changeset	366 self.sentbytescount += getattr(h, 'sentbytescount', 0) - oldbytescount
40033 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	367
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	368 try:
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	369 self.parent.requestscount += 1
40379 41506e3b04ee keepalive: use getattr to avoid AttributeErrors when vcr is in use Augie Fackler <augie@google.com> parents: 40044 diff changeset	370 self.parent.sentbytescount += (
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	371 getattr(h, 'sentbytescount', 0) - oldbytescount
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	372 )
40033 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	373 except AttributeError:
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	374 pass
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	375
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	376
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	377 class HTTPHandler(KeepAliveHandler, urlreq.httphandler):
5983 6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4026 diff changeset	378 pass
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4026 diff changeset	379
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	380
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	381 class HTTPResponse(httplib.HTTPResponse):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	382 # we need to subclass HTTPResponse in order to
51873 34eb3a711955 http: simplify Joerg Sonnenberger <joerg@bec.de> parents: 51871 diff changeset	383 # 1) add close_connection() method
34eb3a711955 http: simplify Joerg Sonnenberger <joerg@bec.de> parents: 51871 diff changeset	384 # 2) add geturl() method
51871 c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	385 # 3) add accounting for read(), readlines() and readinto()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	386
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	387 def __init__(self, sock, debuglevel=0, strict=0, method=None):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	388 httplib.HTTPResponse.__init__(
48986 62baa61efe8f keepalive: remove Python 2 support code Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	389 self, sock, debuglevel=debuglevel, method=method
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	390 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	391 self.fileno = sock.fileno
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	392 self.code = None
40034 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40033 diff changeset	393 self.receivedbytescount = 0
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	394 self._rbuf = b''
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	395 self._rbufsize = 8096
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	396 self._handler = None # inserted by the handler later
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	397 self._host = None # (same)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	398 self._url = None # (same)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	399 self._connection = None # (same)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	400
41463 44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41431 diff changeset	401 # Python 2.7 has a single close() which closes the socket handle.
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41431 diff changeset	402 # This method was effectively renamed to _close_conn() in Python 3. But
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41431 diff changeset	403 # there is also a close(). _close_conn() is called by methods like
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41431 diff changeset	404 # read().
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41431 diff changeset	405
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	406 def close(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	407 if self.fp:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	408 self.fp.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	409 self.fp = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	410 if self._handler:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	411 self._handler._request_closed(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	412 self, self._host, self._connection
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	413 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	414
41463 44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41431 diff changeset	415 def _close_conn(self):
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41431 diff changeset	416 self.close()
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41431 diff changeset	417
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	418 def close_connection(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	419 self._handler._remove_connection(self._host, self._connection, close=1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	420 self.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	421
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	422 def geturl(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	423 return self._url
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	424
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	425 def read(self, amt=None):
51871 c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	426 data = super().read(amt)
40034 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40033 diff changeset	427 self.receivedbytescount += len(data)
51873 34eb3a711955 http: simplify Joerg Sonnenberger <joerg@bec.de> parents: 51871 diff changeset	428 if self._connection is not None:
40414 588f1e9a4d16 http: work around custom http client classes that refuse extra attrs Augie Fackler <augie@google.com> parents: 40379 diff changeset	429 self._connection.receivedbytescount += len(data)
51873 34eb3a711955 http: simplify Joerg Sonnenberger <joerg@bec.de> parents: 51871 diff changeset	430 if self._handler is not None:
40034 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40033 diff changeset	431 self._handler.parent.receivedbytescount += len(data)
51871 c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	432 return data
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	433
30687 5d06f6b73a57 keepalive: remove limit argument from readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30686 diff changeset	434 def readline(self):
51871 c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	435 data = super().readline()
c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	436 self.receivedbytescount += len(data)
51873 34eb3a711955 http: simplify Joerg Sonnenberger <joerg@bec.de> parents: 51871 diff changeset	437 if self._connection is not None:
51871 c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	438 self._connection.receivedbytescount += len(data)
51873 34eb3a711955 http: simplify Joerg Sonnenberger <joerg@bec.de> parents: 51871 diff changeset	439 if self._handler is not None:
51871 c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	440 self._handler.parent.receivedbytescount += len(data)
c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	441 return data
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	442
37297 97eedbd5a56c keepalive: implement readinto() Gregory Szorc <gregory.szorc@gmail.com> parents: 37122 diff changeset	443 def readinto(self, dest):
51871 c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	444 got = super().readinto(dest)
40034 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40033 diff changeset	445 self.receivedbytescount += got
51873 34eb3a711955 http: simplify Joerg Sonnenberger <joerg@bec.de> parents: 51871 diff changeset	446 if self._connection is not None:
51871 c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	447 self._connection.receivedbytescount += got
51873 34eb3a711955 http: simplify Joerg Sonnenberger <joerg@bec.de> parents: 51871 diff changeset	448 if self._handler is not None:
51871 c1ed5ee2ad82 http: reuse Python's implementation of read/readline/readinto Joerg Sonnenberger <joerg@bec.de> parents: 51869 diff changeset	449 self._handler.parent.receivedbytescount += got
37587 192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	450 return got
37297 97eedbd5a56c keepalive: implement readinto() Gregory Szorc <gregory.szorc@gmail.com> parents: 37122 diff changeset	451
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	452
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	453 def safesend(self, str):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	454 """Send `str' to the server.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	455
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	456 Shamelessly ripped off from httplib to patch a bad behavior.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	457 """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	458 # _broken_pipe_resp is an attribute we set in this function
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	459 # if the socket is closed while we're sending data but
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	460 # the server sent us a response before hanging up.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	461 # In that case, we want to pretend to send the rest of the
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	462 # outgoing data, and then let the user use getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	463 # (which we wrap) to get this last response before
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	464 # opening a new socket.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	465 if getattr(self, '_broken_pipe_resp', None) is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	466 return
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	467
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	468 if self.sock is None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	469 if self.auto_open:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	470 self.connect()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	471 else:
16687 e34106fa0dc3 cleanup: "raise SomeException()" -> "raise SomeException" Brodie Rao <brodie@sf.io> parents: 16686 diff changeset	472 raise httplib.NotConnected
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	473
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	474 # send the data to the server. if we get a broken pipe, then close
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	475 # the socket. we want to reconnect when somebody tries to send again.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	476 #
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	477 # NOTE: we DO propagate the error, though, because we cannot simply
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	478 # ignore the error... the caller will know if they can retry.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	479 if self.debuglevel > 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	480 print(b"send:", repr(str))
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	481 try:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	482 blocksize = 8192
14958 fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	483 read = getattr(str, 'read', None)
fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	484 if read is not None:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	485 if self.debuglevel > 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	486 print(b"sending a read()able")
14958 fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	487 data = read(blocksize)
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	488 while data:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	489 self.sock.sendall(data)
40033 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	490 self.sentbytescount += len(data)
14958 fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	491 data = read(blocksize)
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	492 else:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	493 self.sock.sendall(str)
40033 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	494 self.sentbytescount += len(str)
49312 48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 49037 diff changeset	495 except BrokenPipeError:
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 49037 diff changeset	496 if self._HTTPConnection__state == httplib._CS_REQ_SENT:
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 49037 diff changeset	497 self._broken_pipe_resp = None
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 49037 diff changeset	498 self._broken_pipe_resp = self.getresponse()
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 49037 diff changeset	499 reraise = False
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 49037 diff changeset	500 else:
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 49037 diff changeset	501 reraise = True
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 49037 diff changeset	502 self.close()
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	503 if reraise:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	504 raise
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	505
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	506
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	507 def wrapgetresponse(cls):
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45207 diff changeset	508 """Wraps getresponse in cls with a broken-pipe sane version."""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	509
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	510 def safegetresponse(self):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	511 # In safesend() we might set the _broken_pipe_resp
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	512 # attribute, in which case the socket has already
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	513 # been closed and we just need to give them the response
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	514 # back. Otherwise, we use the normal response path.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	515 r = getattr(self, '_broken_pipe_resp', None)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	516 if r is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	517 return r
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	518 return cls.getresponse(self)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	519
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	520 safegetresponse.__doc__ = cls.getresponse.__doc__
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	521 return safegetresponse
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	522
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	523
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	524 class HTTPConnection(httplib.HTTPConnection):
40032 f2dffa1359c6 url: have httpsconnection inherit from our custom HTTPConnection Gregory Szorc <gregory.szorc@gmail.com> parents: 39821 diff changeset	525 # url.httpsconnection inherits from this. So when adding/removing
f2dffa1359c6 url: have httpsconnection inherit from our custom HTTPConnection Gregory Szorc <gregory.szorc@gmail.com> parents: 39821 diff changeset	526 # attributes, be sure to audit httpsconnection() for unintended
f2dffa1359c6 url: have httpsconnection inherit from our custom HTTPConnection Gregory Szorc <gregory.szorc@gmail.com> parents: 39821 diff changeset	527 # consequences.
f2dffa1359c6 url: have httpsconnection inherit from our custom HTTPConnection Gregory Szorc <gregory.szorc@gmail.com> parents: 39821 diff changeset	528
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	529 # use the modified response class
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	530 response_class = HTTPResponse
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	531 send = safesend
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	532 getresponse = wrapgetresponse(httplib.HTTPConnection)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	533
40033 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	534 def __init__(self, args, *kwargs):
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	535 httplib.HTTPConnection.__init__(self, args, *kwargs)
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	536 self.sentbytescount = 0
40034 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40033 diff changeset	537 self.receivedbytescount = 0
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	538
49535 76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	539 def __repr__(self):
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	540 base = super(HTTPConnection, self).__repr__()
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	541 local = "(unconnected)"
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	542 s = self.sock
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	543 if s:
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	544 try:
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	545 local = "%s:%d" % s.getsockname()
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	546 except OSError:
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	547 pass # Likely not connected
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	548 return "<%s: %s <--> %s:%d>" % (base, local, self.host, self.port)
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49534 diff changeset	549
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	550
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	551 #########################################################################
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	552 ##### TEST FUNCTIONS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	553 #########################################################################
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	554
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	555
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	556 def continuity(url):
29341 0d83ad967bf8 cleanup: replace uses of util.(md5\|sha1\|sha256\|sha512) with hashlib.\1 Augie Fackler <raf@durin42.com> parents: 28883 diff changeset	557 md5 = hashlib.md5
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	558 format = b'%25s: %s'
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	559
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	560 # first fetch the file with the normal http handler
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	561 opener = urlreq.buildopener()
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	562 urlreq.installopener(opener)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	563 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	564 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	565 fo.close()
22505 232d437af120 keepalive: fix how md5 is used Mike Hommey <mh@glandium.org> parents: 19872 diff changeset	566 m = md5(foo)
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 45957 diff changeset	567 print(format % (b'normal urllib', hex(m.digest())))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	568
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	569 # now install the keepalive handler and try again
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	570 opener = urlreq.buildopener(HTTPHandler())
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	571 urlreq.installopener(opener)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	572
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	573 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	574 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	575 fo.close()
22505 232d437af120 keepalive: fix how md5 is used Mike Hommey <mh@glandium.org> parents: 19872 diff changeset	576 m = md5(foo)
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 45957 diff changeset	577 print(format % (b'keepalive read', hex(m.digest())))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	578
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	579 fo = urlreq.urlopen(url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	580 foo = b''
14494 1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression Martin Geisler <mg@lazybytes.net> parents: 10394 diff changeset	581 while True:
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	582 f = fo.readline()
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	583 if f:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	584 foo = foo + f
34436 5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34428 diff changeset	585 else:
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34428 diff changeset	586 break
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	587 fo.close()
22505 232d437af120 keepalive: fix how md5 is used Mike Hommey <mh@glandium.org> parents: 19872 diff changeset	588 m = md5(foo)
46114 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 45957 diff changeset	589 print(format % (b'keepalive readline', hex(m.digest())))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	590
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	591
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	592 def comp(N, url):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	593 print(b' making %i connections to:\n %s' % (N, url))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	594
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	595 procutil.stdout.write(b' first using the normal urllib handlers')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	596 # first use normal opener
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	597 opener = urlreq.buildopener()
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	598 urlreq.installopener(opener)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	599 t1 = fetch(N, url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	600 print(b' TIME: %.3f s' % t1)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	601
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	602 procutil.stdout.write(b' now using the keepalive handler ')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	603 # now install the keepalive handler and try again
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	604 opener = urlreq.buildopener(HTTPHandler())
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	605 urlreq.installopener(opener)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	606 t2 = fetch(N, url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	607 print(b' TIME: %.3f s' % t2)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	608 print(b' improvement factor: %.2f' % (t1 / t2))
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	609
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	610
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	611 def fetch(N, url, delay=0):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	612 import time
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	613
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	614 lens = []
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	615 starttime = time.time()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	616 for i in range(N):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	617 if delay and i > 0:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	618 time.sleep(delay)
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	619 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	620 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	621 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	622 lens.append(len(foo))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	623 diff = time.time() - starttime
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	624
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	625 j = 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	626 for i in lens[1:]:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	627 j = j + 1
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	628 if not i == lens[0]:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	629 print(b"WARNING: inconsistent length on read %i: %i" % (j, i))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	630
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	631 return diff
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	632
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	633
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	634 def test_timeout(url):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	635 global DEBUG
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	636 dbbackup = DEBUG
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	637
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 49015 diff changeset	638 class FakeLogger:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	639 def debug(self, msg, *args):
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	640 print(msg % args)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	641
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	642 info = warning = error = debug
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	643
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	644 DEBUG = FakeLogger()
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	645 print(b" fetching the file to establish a connection")
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	646 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	647 data1 = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	648 fo.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	649
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	650 i = 20
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	651 print(b" waiting %i seconds for the server to close the connection" % i)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	652 while i > 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	653 procutil.stdout.write(b'\r %2i' % i)
37122 d4a2e0d5d042 procutil: bulk-replace util.std* to point to new module Yuya Nishihara <yuya@tcha.org> parents: 36328 diff changeset	654 procutil.stdout.flush()
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	655 time.sleep(1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	656 i -= 1
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	657 procutil.stderr.write(b'\r')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	658
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	659 print(b" fetching the file a second time")
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	660 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	661 data2 = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	662 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	663
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	664 if data1 == data2:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	665 print(b' data are identical')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	666 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	667 print(b' ERROR: DATA DIFFER')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	668
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	669 DEBUG = dbbackup
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	670
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	671
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	672 def test(url, N=10):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	673 print(b"performing continuity test (making sure stuff isn't corrupted)")
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	674 continuity(url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	675 print(b'')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	676 print(b"performing speed comparison")
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	677 comp(N, url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	678 print(b'')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	679 print(b"performing dropped-connection check")
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	680 test_timeout(url)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	681
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	682
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	683 if __name__ == '__main__':
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	684 import time
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41463 diff changeset	685
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	686 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	687 N = int(sys.argv[1])
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	688 url = sys.argv[2]
16688 cfb6682961b8 cleanup: replace naked excepts with more specific ones Brodie Rao <brodie@sf.io> parents: 16687 diff changeset	689 except (IndexError, ValueError):
51677 bd490cdf764a py3: use str literal instead of bytes literal Manuel Jacob <me@manueljacob.de> parents: 50952 diff changeset	690 print("%s <integer> <url>" % sys.argv[0])
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	691 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	692 test(url, N)

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/keepalive.py @ 51875:454feddab720