mercurial-scm/hg: mercurial/keepalive.py annotate

annotate mercurial/keepalive.py @ 51528:f85f23f1479b

branchcache: skip entries that are topological heads in the on disk file In the majority of cases, topological heads are also branch heads. We have efficient way to get the topological heads and efficient way to retrieve their branch information. So there is little value in putting them in the branch cache file explicitly. On the contrary, writing them explicitly tend to create very large cache file that are inefficient to read and update. So the branch cache v3 format is no longer including them. This changeset focus on the format aspect and have no focus on the performance aspect. We will cover that later.

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Thu, 07 Mar 2024 10:55:22 +0100
parents	18c8c18993f0
children	bd490cdf764a ace0da86edd0

rev	line source
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	1 # This library is free software; you can redistribute it and/or
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	2 # modify it under the terms of the GNU Lesser General Public
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	3 # License as published by the Free Software Foundation; either
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	4 # version 2.1 of the License, or (at your option) any later version.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	5 #
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	6 # This library is distributed in the hope that it will be useful,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	7 # but WITHOUT ANY WARRANTY; without even the implied warranty of
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	9 # Lesser General Public License for more details.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	10 #
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	11 # You should have received a copy of the GNU Lesser General Public
15782 7de7630053cb Remove FSF mailing address from GPL headers Martin Geisler <mg@aragost.com> parents: 14958 diff changeset	12 # License along with this library; if not, see
7de7630053cb Remove FSF mailing address from GPL headers Martin Geisler <mg@aragost.com> parents: 14958 diff changeset	13 # <http://www.gnu.org/licenses/>.
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	14
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	15 # This file is part of urlgrabber, a high-level cross-protocol url-grabber
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	16 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	17
4026 8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	18 # Modified by Benoit Boissinot:
8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	19 # - fix for digest auth (inspired from urllib2.py @ Python v2.4)
6470 ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions Dirkjan Ochtman <dirkjan@ochtman.nl> parents: 6001 diff changeset	20 # Modified by Dirkjan Ochtman:
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions Dirkjan Ochtman <dirkjan@ochtman.nl> parents: 6001 diff changeset	21 # - import md5 function from a local util module
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	22 # Modified by Augie Fackler:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	23 # - add safesend method and use it to prevent broken pipe errors
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	24 # on large POST requests
4026 8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	25
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	26 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	27
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	28 >>> import urllib2
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	29 >>> from keepalive import HTTPHandler
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	30 >>> keepalive_handler = HTTPHandler()
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	31 >>> opener = urlreq.buildopener(keepalive_handler)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	32 >>> urlreq.installopener(opener)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	33 >>>
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	34 >>> fo = urlreq.urlopen('http://www.python.org')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	35
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	36 If a connection to a given host is requested, and all of the existing
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	37 connections are still in use, another connection will be opened. If
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	38 the handler tries to use an existing connection but it fails in some
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	39 way, it will be closed and removed from the pool.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	40
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	41 To remove the handler, simply re-run build_opener with no arguments, and
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	42 install that opener.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	43
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	44 You can explicitly close connections by using the close_connection()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	45 method of the returned file-like object (described below) or you can
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	46 use the handler methods:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	47
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	48 close_connection(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	49 close_all()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	50 open_connections()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	51
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	52 NOTE: using the close_connection and close_all methods of the handler
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	53 should be done with care when using multiple threads.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	54 * there is nothing that prevents another thread from creating new
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	55 connections immediately after connections are closed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	56 * no checks are done to prevent in-use connections from being closed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	57
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	58 >>> keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	59
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	60 EXTRA ATTRIBUTES AND METHODS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	61
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	62 Upon a status of 200, the object returned has a few additional
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	63 attributes and methods, which should not be used if you want to
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	64 remain consistent with the normal urllib2-returned objects:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	65
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	66 close_connection() - close the connection to the host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	67 readlines() - you know, readlines()
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	68 status - the return status (i.e. 404)
e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	69 reason - english translation of status (i.e. 'File not found')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	70
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	71 If you want the best of both worlds, use this inside an
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	72 AttributeError-catching try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	73
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	74 >>> try: status = fo.status
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	75 >>> except AttributeError: status = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	76
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	77 Unfortunately, these are ONLY there if status == 200, so it's not
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	78 easy to distinguish between non-200 responses. The reason is that
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	79 urllib2 tries to do clever things with error codes 301, 302, 401,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	80 and 407, and it wraps the object upon return.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	81 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	82
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	83 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	84
27507 a16489f9132d keepalive: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	85
41408 a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40874 diff changeset	86 import collections
29341 0d83ad967bf8 cleanup: replace uses of util.(md5\|sha1\|sha256\|sha512) with hashlib.\1 Augie Fackler <raf@durin42.com> parents: 28883 diff changeset	87 import hashlib
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	88 import socket
27507 a16489f9132d keepalive: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	89 import sys
29456 e61d384e3be4 keepalive: switch from thread to threading module Pulkit Goyal <7895pulkit@gmail.com> parents: 29455 diff changeset	90 import threading
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	91
34308 9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	92 from .i18n import _
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 45942 diff changeset	93 from .node import hex
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	94 from . import (
34427 a454123f5d94 keepalive: python 3 portability tweaks Augie Fackler <augie@google.com> parents: 34331 diff changeset	95 pycompat,
34466 1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34435 diff changeset	96 urllibcompat,
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	97 util,
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	98 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	99 from .utils import procutil
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	100
29455 0c741fd6158a py3: conditionalize httplib import Pulkit Goyal <7895pulkit@gmail.com> parents: 29341 diff changeset	101 httplib = util.httplib
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	102 urlerr = util.urlerr
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	103 urlreq = util.urlreq
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	104
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	105 DEBUG = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	106
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	107
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48924 diff changeset	108 class ConnectionManager:
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	109 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	110 The connection manager must be able to:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	111 * keep track of all existing
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45352 diff changeset	112 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	113
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	114 def __init__(self):
29456 e61d384e3be4 keepalive: switch from thread to threading module Pulkit Goyal <7895pulkit@gmail.com> parents: 29455 diff changeset	115 self._lock = threading.Lock()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	116 self._hostmap = collections.defaultdict(list) # host -> [connection]
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	117 self._connmap = {} # map connections to host
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	118 self._readymap = {} # map connection to ready state
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	119
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	120 def add(self, host, connection, ready):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	121 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	122 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	123 self._hostmap[host].append(connection)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	124 self._connmap[connection] = host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	125 self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	126 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	127 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	128
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	129 def remove(self, connection):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	130 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	131 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	132 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	133 host = self._connmap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	134 except KeyError:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	135 pass
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	136 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	137 del self._connmap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	138 del self._readymap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	139 self._hostmap[host].remove(connection)
34435 5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34427 diff changeset	140 if not self._hostmap[host]:
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34427 diff changeset	141 del self._hostmap[host]
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	142 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	143 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	144
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	145 def set_ready(self, connection, ready):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	146 try:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	147 self._readymap[connection] = ready
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	148 except KeyError:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	149 pass
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	150
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	151 def get_ready_conn(self, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	152 conn = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	153 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	154 try:
41408 a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40874 diff changeset	155 for c in self._hostmap[host]:
a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40874 diff changeset	156 if self._readymap[c]:
41409 1db94ebbc207 keepalive: track ready state with a bool Gregory Szorc <gregory.szorc@gmail.com> parents: 41408 diff changeset	157 self._readymap[c] = False
41408 a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40874 diff changeset	158 conn = c
a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40874 diff changeset	159 break
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	160 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	161 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	162 return conn
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	163
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	164 def get_all(self, host=None):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	165 if host:
41408 a43acfa2b76d keepalive: use collections.defaultdict for host map Gregory Szorc <gregory.szorc@gmail.com> parents: 40874 diff changeset	166 return list(self._hostmap[host])
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	167 else:
49542 8251f7cc787d keepalive: ensure `close_all()` actually closes all cached connections Matt Harbison <matt_harbison@yahoo.com> parents: 49304 diff changeset	168 return dict(
8251f7cc787d keepalive: ensure `close_all()` actually closes all cached connections Matt Harbison <matt_harbison@yahoo.com> parents: 49304 diff changeset	169 {h: list(conns) for (h, conns) in self._hostmap.items()}
8251f7cc787d keepalive: ensure `close_all()` actually closes all cached connections Matt Harbison <matt_harbison@yahoo.com> parents: 49304 diff changeset	170 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	171
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	172
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48924 diff changeset	173 class KeepAliveHandler:
40043 6509fcec830c url: allow to configure timeout on http connection C?dric Krier <ced@b2ck.com> parents: 40033 diff changeset	174 def __init__(self, timeout=None):
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	175 self._cm = ConnectionManager()
40043 6509fcec830c url: allow to configure timeout on http connection C?dric Krier <ced@b2ck.com> parents: 40033 diff changeset	176 self._timeout = timeout
40032 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	177 self.requestscount = 0
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	178 self.sentbytescount = 0
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	179
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	180 #### Connection Management
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	181 def open_connections(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	182 """return a list of connected hosts and the number of connections
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	183 to each. [('foo.com:80', 2), ('bar.org', 1)]"""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	184 return [(host, len(li)) for (host, li) in self._cm.get_all().items()]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	185
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	186 def close_connection(self, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	187 """close connection(s) to <host>
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	188 host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	189 no error occurs if there is no connection to that host."""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	190 for h in self._cm.get_all(host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	191 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	192 h.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	193
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	194 def close_all(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	195 """close all open connections"""
48924 dea766fca7e1 keepalive: remove pycompat.iteritems() Gregory Szorc <gregory.szorc@gmail.com> parents: 48895 diff changeset	196 for host, conns in self._cm.get_all().items():
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	197 for h in conns:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	198 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	199 h.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	200
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	201 def _request_closed(self, request, host, connection):
17251 98166640b356 help: fix some instances of 'the the' Mads Kiilerich <mads@kiilerich.com> parents: 16705 diff changeset	202 """tells us that this request is now closed and that the
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	203 connection is ready for another request"""
41409 1db94ebbc207 keepalive: track ready state with a bool Gregory Szorc <gregory.szorc@gmail.com> parents: 41408 diff changeset	204 self._cm.set_ready(connection, True)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	205
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	206 def _remove_connection(self, host, connection, close=0):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	207 if close:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	208 connection.close()
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	209 self._cm.remove(connection)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	210
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	211 #### Transaction Execution
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	212 def http_open(self, req):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	213 return self.do_open(HTTPConnection, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	214
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	215 def do_open(self, http_class, req):
34466 1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34435 diff changeset	216 host = urllibcompat.gethost(req)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	217 if not host:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	218 raise urlerr.urlerror(b'no host given')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	219
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	220 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	221 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	222 while h:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	223 r = self._reuse_connection(h, req, host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	224
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	225 # if this response is non-None, then it worked and we're
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	226 # done. Break out, skipping the else block.
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	227 if r:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	228 break
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	229
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	230 # connection is bad - possibly closed by server
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	231 # discard it and ask for the next free connection
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	232 h.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	233 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	234 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	235 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	236 # no (working) free connections were found. Create a new one.
40043 6509fcec830c url: allow to configure timeout on http connection C?dric Krier <ced@b2ck.com> parents: 40033 diff changeset	237 h = http_class(host, timeout=self._timeout)
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	238 if DEBUG:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	239 DEBUG.info(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	240 b"creating new connection to %s (%d)", host, id(h)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	241 )
41409 1db94ebbc207 keepalive: track ready state with a bool Gregory Szorc <gregory.szorc@gmail.com> parents: 41408 diff changeset	242 self._cm.add(host, h, False)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	243 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	244 r = h.getresponse()
34308 9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	245 # The string form of BadStatusLine is the status line. Add some context
9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	246 # to make the error message slightly more useful.
9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	247 except httplib.BadStatusLine as err:
34427 a454123f5d94 keepalive: python 3 portability tweaks Augie Fackler <augie@google.com> parents: 34331 diff changeset	248 raise urlerr.urlerror(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	249 _(b'bad HTTP status line: %s') % pycompat.sysbytes(err.line)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	250 )
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 22958 diff changeset	251 except (socket.error, httplib.HTTPException) as err:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	252 raise urlerr.urlerror(err)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	253
39649 d6d094259d9c keepalive: work around slight deficiency in vcr Augie Fackler <raf@durin42.com> parents: 37688 diff changeset	254 # If not a persistent connection, don't try to reuse it. Look
d6d094259d9c keepalive: work around slight deficiency in vcr Augie Fackler <raf@durin42.com> parents: 37688 diff changeset	255 # for this using getattr() since vcr doesn't define this
d6d094259d9c keepalive: work around slight deficiency in vcr Augie Fackler <raf@durin42.com> parents: 37688 diff changeset	256 # attribute, and in that case always close the connection.
43103 c95b2f40db7c py3: stop normalizing 2nd argument of attr() to unicode Gregory Szorc <gregory.szorc@gmail.com>* parents: 43089 diff changeset	257 if getattr(r, 'will_close', True):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	258 self._cm.remove(h)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	259
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	260 if DEBUG:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	261 DEBUG.info(b"STATUS: %s, %s", r.status, r.reason)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	262 r._handler = self
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	263 r._host = host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	264 r._url = req.get_full_url()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	265 r._connection = h
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	266 r.code = r.status
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	267 r.headers = r.msg
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	268 r.msg = r.reason
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	269
30487 88a448a12ae8 keepalive: discard legacy Python support for error handling Augie Fackler <augie@google.com> parents: 30473 diff changeset	270 return r
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	271
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	272 def _reuse_connection(self, h, req, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	273 """start the transaction with a re-used connection
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	274 return a response object (r) upon success or None on failure.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	275 This DOES not close or remove bad connections in cases where
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	276 it returns. However, if an unexpected exception occurs, it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	277 will close and remove the connection before re-raising.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	278 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	279 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	280 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	281 r = h.getresponse()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	282 # note: just because we got something back doesn't mean it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	283 # worked. We'll check the version below, too.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	284 except (socket.error, httplib.HTTPException):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	285 r = None
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	286 except: # re-raises
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	287 # adding this block just in case we've missed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	288 # something we will still raise the exception, but
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	289 # lets try and close the connection and remove it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	290 # first. We previously got into a nasty loop
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	291 # where an exception was uncaught, and so the
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	292 # connection stayed open. On the next try, the
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	293 # same exception was raised, etc. The trade-off is
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	294 # that it's now possible this call will raise
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	295 # a DIFFERENT exception
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	296 if DEBUG:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	297 DEBUG.error(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43106 diff changeset	298 b"unexpected exception - closing connection to %s (%d)",
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	299 host,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	300 id(h),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	301 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	302 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	303 h.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	304 raise
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	305
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	306 if r is None or r.version == 9:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	307 # httplib falls back to assuming HTTP 0.9 if it gets a
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	308 # bad header back. This is most likely to happen if
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	309 # the socket has been closed by the server since we
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	310 # last used the connection.
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	311 if DEBUG:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	312 DEBUG.info(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	313 b"failed to re-use connection to %s (%d)", host, id(h)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	314 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	315 r = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	316 else:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	317 if DEBUG:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	318 DEBUG.info(b"re-using connection to %s (%d)", host, id(h))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	319
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	320 return r
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	321
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	322 def _start_transaction(self, h, req):
40379 41506e3b04ee keepalive: use getattr to avoid AttributeErrors when vcr is in use Augie Fackler <augie@google.com> parents: 40043 diff changeset	323 oldbytescount = getattr(h, 'sentbytescount', 0)
40032 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	324
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	325 # What follows mostly reimplements HTTPConnection.request()
31999 aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30922 diff changeset	326 # except it adds self.parent.addheaders in the mix and sends headers
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30922 diff changeset	327 # in a deterministic order (to make testing easier).
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30922 diff changeset	328 headers = util.sortdict(self.parent.addheaders)
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30922 diff changeset	329 headers.update(sorted(req.headers.items()))
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30922 diff changeset	330 headers.update(sorted(req.unredirected_hdrs.items()))
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30922 diff changeset	331 headers = util.sortdict((n.lower(), v) for n, v in headers.items())
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	332 skipheaders = {}
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	333 for n in ('host', 'accept-encoding'):
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	334 if n in headers:
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	335 skipheaders['skip_' + n.replace('-', '_')] = 1
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	336 try:
34466 1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34435 diff changeset	337 if urllibcompat.hasdata(req):
1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34435 diff changeset	338 data = urllibcompat.getdata(req)
30922 1beeb5185930 keepalive: honor urllib2 style get_method overrides John Mulligan <phlogistonjohn@asynchrono.us> parents: 30688 diff changeset	339 h.putrequest(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	340 req.get_method(),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	341 urllibcompat.getselector(req),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	342 **skipheaders
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	343 )
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	344 if 'content-type' not in headers:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	345 h.putheader(
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	346 'Content-type', 'application/x-www-form-urlencoded'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	347 )
43506 9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	348 if 'content-length' not in headers:
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings Augie Fackler <augie@google.com> parents: 43117 diff changeset	349 h.putheader('Content-length', '%d' % len(data))
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	350 else:
30922 1beeb5185930 keepalive: honor urllib2 style get_method overrides John Mulligan <phlogistonjohn@asynchrono.us> parents: 30688 diff changeset	351 h.putrequest(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	352 req.get_method(),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	353 urllibcompat.getselector(req),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	354 **skipheaders
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	355 )
28278 b1b22185c764 keepalive: remove useless parentheses around exception type Gregory Szorc <gregory.szorc@gmail.com> parents: 27616 diff changeset	356 except socket.error as err:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	357 raise urlerr.urlerror(err)
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	358 for k, v in headers.items():
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	359 h.putheader(k, v)
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	360 h.endheaders()
34466 1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects Augie Fackler <augie@google.com> parents: 34435 diff changeset	361 if urllibcompat.hasdata(req):
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	362 h.send(data)
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	363
40032 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	364 # This will fail to record events in case of I/O failure. That's OK.
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	365 self.requestscount += 1
40379 41506e3b04ee keepalive: use getattr to avoid AttributeErrors when vcr is in use Augie Fackler <augie@google.com> parents: 40043 diff changeset	366 self.sentbytescount += getattr(h, 'sentbytescount', 0) - oldbytescount
40032 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	367
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	368 try:
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	369 self.parent.requestscount += 1
40379 41506e3b04ee keepalive: use getattr to avoid AttributeErrors when vcr is in use Augie Fackler <augie@google.com> parents: 40043 diff changeset	370 self.parent.sentbytescount += (
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	371 getattr(h, 'sentbytescount', 0) - oldbytescount
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	372 )
40032 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	373 except AttributeError:
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	374 pass
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	375
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	376
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	377 class HTTPHandler(KeepAliveHandler, urlreq.httphandler):
5983 6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4026 diff changeset	378 pass
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4026 diff changeset	379
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	380
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	381 class HTTPResponse(httplib.HTTPResponse):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	382 # we need to subclass HTTPResponse in order to
37297 97eedbd5a56c keepalive: implement readinto() Gregory Szorc <gregory.szorc@gmail.com> parents: 37119 diff changeset	383 # 1) add readline(), readlines(), and readinto() methods
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	384 # 2) add close_connection() methods
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	385 # 3) add info() and geturl() methods
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	386
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	387 # in order to add readline(), read must be modified to deal with a
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	388 # buffer. example: readline must read a buffer and then spit back
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	389 # one line at a time. The only real alternative is to read one
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	390 # BYTE at a time (ick). Once something has been read, it can't be
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	391 # put back (ok, maybe it can, but that's even uglier than this),
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	392 # so if you THEN do a normal read, you must first take stuff from
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	393 # the buffer.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	394
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	395 # the read method wraps the original to accommodate buffering,
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	396 # although read() never adds to the buffer.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	397 # Both readline and readlines have been stolen with almost no
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	398 # modification from socket.py
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	399
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	400 def __init__(self, sock, debuglevel=0, strict=0, method=None):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	401 httplib.HTTPResponse.__init__(
48895 62baa61efe8f keepalive: remove Python 2 support code Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	402 self, sock, debuglevel=debuglevel, method=method
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	403 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	404 self.fileno = sock.fileno
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	405 self.code = None
40033 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	406 self.receivedbytescount = 0
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	407 self._rbuf = b''
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	408 self._rbufsize = 8096
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	409 self._handler = None # inserted by the handler later
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	410 self._host = None # (same)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	411 self._url = None # (same)
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	412 self._connection = None # (same)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	413
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	414 _raw_read = httplib.HTTPResponse.read
37587 192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	415 _raw_readinto = getattr(httplib.HTTPResponse, 'readinto', None)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	416
41441 44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41409 diff changeset	417 # Python 2.7 has a single close() which closes the socket handle.
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41409 diff changeset	418 # This method was effectively renamed to _close_conn() in Python 3. But
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41409 diff changeset	419 # there is also a close(). _close_conn() is called by methods like
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41409 diff changeset	420 # read().
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41409 diff changeset	421
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	422 def close(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	423 if self.fp:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	424 self.fp.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	425 self.fp = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	426 if self._handler:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	427 self._handler._request_closed(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	428 self, self._host, self._connection
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	429 )
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	430
41441 44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41409 diff changeset	431 def _close_conn(self):
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41409 diff changeset	432 self.close()
44d752efdbce keepalive: implement _close_conn() so closes are known Gregory Szorc <gregory.szorc@gmail.com> parents: 41409 diff changeset	433
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	434 def close_connection(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	435 self._handler._remove_connection(self._host, self._connection, close=1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	436 self.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	437
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	438 def info(self):
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	439 return self.headers
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	440
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	441 def geturl(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	442 return self._url
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	443
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	444 def read(self, amt=None):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	445 # the _rbuf test is only in this first if for speed. It's not
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	446 # logically necessary
34331 531332502568 style: always use `x is not None` instead of `not x is None` Alex Gaynor <agaynor@mozilla.com> parents: 34308 diff changeset	447 if self._rbuf and amt is not None:
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	448 L = len(self._rbuf)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	449 if amt > L:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	450 amt -= L
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	451 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	452 s = self._rbuf[:amt]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	453 self._rbuf = self._rbuf[amt:]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	454 return s
39816 1cf1680b0554 keepalive: be more careful about self._rbuf when calling super impls Augie Fackler <augie@google.com> parents: 39649 diff changeset	455 # Careful! http.client.HTTPResponse.read() on Python 3 is
1cf1680b0554 keepalive: be more careful about self._rbuf when calling super impls Augie Fackler <augie@google.com> parents: 39649 diff changeset	456 # implemented using readinto(), which can duplicate self._rbuf
1cf1680b0554 keepalive: be more careful about self._rbuf when calling super impls Augie Fackler <augie@google.com> parents: 39649 diff changeset	457 # if it's not empty.
1cf1680b0554 keepalive: be more careful about self._rbuf when calling super impls Augie Fackler <augie@google.com> parents: 39649 diff changeset	458 s = self._rbuf
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	459 self._rbuf = b''
40033 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	460 data = self._raw_read(amt)
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	461
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	462 self.receivedbytescount += len(data)
40426 588f1e9a4d16 http: work around custom http client classes that refuse extra attrs Augie Fackler <augie@google.com> parents: 40379 diff changeset	463 try:
588f1e9a4d16 http: work around custom http client classes that refuse extra attrs Augie Fackler <augie@google.com> parents: 40379 diff changeset	464 self._connection.receivedbytescount += len(data)
588f1e9a4d16 http: work around custom http client classes that refuse extra attrs Augie Fackler <augie@google.com> parents: 40379 diff changeset	465 except AttributeError:
588f1e9a4d16 http: work around custom http client classes that refuse extra attrs Augie Fackler <augie@google.com> parents: 40379 diff changeset	466 pass
40033 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	467 try:
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	468 self._handler.parent.receivedbytescount += len(data)
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	469 except AttributeError:
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	470 pass
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	471
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	472 s += data
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	473 return s
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	474
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	475 # stolen from Python SVN #68532 to fix issue1088
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	476 def _read_chunked(self, amt):
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	477 chunk_left = self.chunk_left
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30487 diff changeset	478 parts = []
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	479
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	480 while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	481 if chunk_left is None:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	482 line = self.fp.readline()
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	483 i = line.find(b';')
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	484 if i >= 0:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	485 line = line[:i] # strip chunk-extensions
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	486 try:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	487 chunk_left = int(line, 16)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	488 except ValueError:
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	489 # close the connection as protocol synchronization is
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	490 # probably lost
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	491 self.close()
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	492 raise httplib.IncompleteRead(b''.join(parts))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	493 if chunk_left == 0:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	494 break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	495 if amt is None:
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30487 diff changeset	496 parts.append(self._safe_read(chunk_left))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	497 elif amt < chunk_left:
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30487 diff changeset	498 parts.append(self._safe_read(amt))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	499 self.chunk_left = chunk_left - amt
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	500 return b''.join(parts)
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	501 elif amt == chunk_left:
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30487 diff changeset	502 parts.append(self._safe_read(amt))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	503 self._safe_read(2) # toss the CRLF at the end of the chunk
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	504 self.chunk_left = None
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	505 return b''.join(parts)
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	506 else:
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30487 diff changeset	507 parts.append(self._safe_read(chunk_left))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	508 amt -= chunk_left
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	509
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	510 # we read the whole chunk, get another
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	511 self._safe_read(2) # toss the CRLF at the end of the chunk
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	512 chunk_left = None
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	513
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	514 # read and discard trailer up to the CRLF terminator
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	515 ### note: we shouldn't have any trailers!
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	516 while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	517 line = self.fp.readline()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	518 if not line:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	519 # a vanishingly small number of sites EOF without
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	520 # sending the trailer
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	521 break
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	522 if line == b'\r\n':
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	523 break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	524
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	525 # we read everything; close the "file"
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	526 self.close()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	527
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	528 return b''.join(parts)
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	529
30687 5d06f6b73a57 keepalive: remove limit argument from readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30686 diff changeset	530 def readline(self):
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	531 # Fast path for a line is already available in read buffer.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	532 i = self._rbuf.find(b'\n')
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	533 if i >= 0:
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	534 i += 1
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	535 line = self._rbuf[:i]
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	536 self._rbuf = self._rbuf[i:]
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	537 return line
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	538
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	539 # No newline in local buffer. Read until we find one.
45352 49f8ba4febec keepalive: Do not append _rbuf if _raw_readinto exists (issue6356) C?dric Krier <ced@b2ck.com> parents: 43506 diff changeset	540 # readinto read via readinto will already return _rbuf
49f8ba4febec keepalive: Do not append _rbuf if _raw_readinto exists (issue6356) C?dric Krier <ced@b2ck.com> parents: 43506 diff changeset	541 if self._raw_readinto is None:
49f8ba4febec keepalive: Do not append _rbuf if _raw_readinto exists (issue6356) C?dric Krier <ced@b2ck.com> parents: 43506 diff changeset	542 chunks = [self._rbuf]
49f8ba4febec keepalive: Do not append _rbuf if _raw_readinto exists (issue6356) C?dric Krier <ced@b2ck.com> parents: 43506 diff changeset	543 else:
49f8ba4febec keepalive: Do not append _rbuf if _raw_readinto exists (issue6356) C?dric Krier <ced@b2ck.com> parents: 43506 diff changeset	544 chunks = []
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	545 i = -1
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	546 readsize = self._rbufsize
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	547 while True:
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	548 new = self._raw_read(readsize)
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	549 if not new:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	550 break
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	551
40033 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	552 self.receivedbytescount += len(new)
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	553 self._connection.receivedbytescount += len(new)
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	554 try:
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	555 self._handler.parent.receivedbytescount += len(new)
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	556 except AttributeError:
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	557 pass
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	558
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	559 chunks.append(new)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	560 i = new.find(b'\n')
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	561 if i >= 0:
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	562 break
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	563
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	564 # We either have exhausted the stream or have a newline in chunks[-1].
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	565
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	566 # EOF
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	567 if i == -1:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	568 self._rbuf = b''
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	569 return b''.join(chunks)
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	570
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	571 i += 1
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	572 self._rbuf = chunks[-1][i:]
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	573 chunks[-1] = chunks[-1][:i]
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	574 return b''.join(chunks)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	575
19872 681f7b9213a4 check-code: check for spaces around = for named parameters Mads Kiilerich <madski@unity3d.com> parents: 17700 diff changeset	576 def readlines(self, sizehint=0):
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	577 total = 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	578 list = []
14494 1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression Martin Geisler <mg@lazybytes.net> parents: 10394 diff changeset	579 while True:
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	580 line = self.readline()
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	581 if not line:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	582 break
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	583 list.append(line)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	584 total += len(line)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	585 if sizehint and total >= sizehint:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	586 break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	587 return list
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	588
37297 97eedbd5a56c keepalive: implement readinto() Gregory Szorc <gregory.szorc@gmail.com> parents: 37119 diff changeset	589 def readinto(self, dest):
37587 192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	590 if self._raw_readinto is None:
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	591 res = self.read(len(dest))
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	592 if not res:
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	593 return 0
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	594 dest[0 : len(res)] = res
37587 192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	595 return len(res)
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	596 total = len(dest)
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	597 have = len(self._rbuf)
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	598 if have >= total:
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	599 dest[0:total] = self._rbuf[:total]
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	600 self._rbuf = self._rbuf[total:]
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	601 return total
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	602 mv = memoryview(dest)
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	603 got = self._raw_readinto(mv[have:total])
40033 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	604
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	605 self.receivedbytescount += got
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	606 self._connection.receivedbytescount += got
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	607 try:
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	608 self._handler.receivedbytescount += got
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	609 except AttributeError:
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	610 pass
5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	611
37587 192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	612 dest[0:have] = self._rbuf
192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	613 got += len(self._rbuf)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	614 self._rbuf = b''
37587 192b7ad06932 keepalive: rewrite readinto() to not use read() Augie Fackler <augie@google.com> parents: 37297 diff changeset	615 return got
37297 97eedbd5a56c keepalive: implement readinto() Gregory Szorc <gregory.szorc@gmail.com> parents: 37119 diff changeset	616
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	617
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	618 def safesend(self, str):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	619 """Send `str' to the server.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	620
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	621 Shamelessly ripped off from httplib to patch a bad behavior.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	622 """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	623 # _broken_pipe_resp is an attribute we set in this function
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	624 # if the socket is closed while we're sending data but
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	625 # the server sent us a response before hanging up.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	626 # In that case, we want to pretend to send the rest of the
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	627 # outgoing data, and then let the user use getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	628 # (which we wrap) to get this last response before
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	629 # opening a new socket.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	630 if getattr(self, '_broken_pipe_resp', None) is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	631 return
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	632
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	633 if self.sock is None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	634 if self.auto_open:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	635 self.connect()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	636 else:
16687 e34106fa0dc3 cleanup: "raise SomeException()" -> "raise SomeException" Brodie Rao <brodie@sf.io> parents: 16686 diff changeset	637 raise httplib.NotConnected
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	638
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	639 # send the data to the server. if we get a broken pipe, then close
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	640 # the socket. we want to reconnect when somebody tries to send again.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	641 #
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	642 # NOTE: we DO propagate the error, though, because we cannot simply
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	643 # ignore the error... the caller will know if they can retry.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	644 if self.debuglevel > 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	645 print(b"send:", repr(str))
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	646 try:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	647 blocksize = 8192
14958 fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	648 read = getattr(str, 'read', None)
fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	649 if read is not None:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	650 if self.debuglevel > 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	651 print(b"sending a read()able")
14958 fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	652 data = read(blocksize)
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	653 while data:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	654 self.sock.sendall(data)
40032 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	655 self.sentbytescount += len(data)
14958 fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	656 data = read(blocksize)
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	657 else:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	658 self.sock.sendall(str)
40032 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	659 self.sentbytescount += len(str)
49304 48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 48946 diff changeset	660 except BrokenPipeError:
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 48946 diff changeset	661 if self._HTTPConnection__state == httplib._CS_REQ_SENT:
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 48946 diff changeset	662 self._broken_pipe_resp = None
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 48946 diff changeset	663 self._broken_pipe_resp = self.getresponse()
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 48946 diff changeset	664 reraise = False
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 48946 diff changeset	665 else:
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 48946 diff changeset	666 reraise = True
48f1b314056b py3: catch BrokenPipeError instead of checking errno == EPIPE Manuel Jacob <me@manueljacob.de> parents: 48946 diff changeset	667 self.close()
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	668 if reraise:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	669 raise
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	670
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	671
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	672 def wrapgetresponse(cls):
45942 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45352 diff changeset	673 """Wraps getresponse in cls with a broken-pipe sane version."""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	674
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	675 def safegetresponse(self):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	676 # In safesend() we might set the _broken_pipe_resp
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	677 # attribute, in which case the socket has already
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	678 # been closed and we just need to give them the response
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	679 # back. Otherwise, we use the normal response path.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	680 r = getattr(self, '_broken_pipe_resp', None)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	681 if r is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	682 return r
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	683 return cls.getresponse(self)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	684
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	685 safegetresponse.__doc__ = cls.getresponse.__doc__
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	686 return safegetresponse
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	687
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	688
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	689 class HTTPConnection(httplib.HTTPConnection):
40031 f2dffa1359c6 url: have httpsconnection inherit from our custom HTTPConnection Gregory Szorc <gregory.szorc@gmail.com> parents: 39816 diff changeset	690 # url.httpsconnection inherits from this. So when adding/removing
f2dffa1359c6 url: have httpsconnection inherit from our custom HTTPConnection Gregory Szorc <gregory.szorc@gmail.com> parents: 39816 diff changeset	691 # attributes, be sure to audit httpsconnection() for unintended
f2dffa1359c6 url: have httpsconnection inherit from our custom HTTPConnection Gregory Szorc <gregory.szorc@gmail.com> parents: 39816 diff changeset	692 # consequences.
f2dffa1359c6 url: have httpsconnection inherit from our custom HTTPConnection Gregory Szorc <gregory.szorc@gmail.com> parents: 39816 diff changeset	693
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	694 # use the modified response class
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	695 response_class = HTTPResponse
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	696 send = safesend
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	697 getresponse = wrapgetresponse(httplib.HTTPConnection)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	698
40032 dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	699 def __init__(self, args, *kwargs):
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	700 httplib.HTTPConnection.__init__(self, args, *kwargs)
dc82ad1b7f77 keepalive: track request count and bytes sent Gregory Szorc <gregory.szorc@gmail.com> parents: 40031 diff changeset	701 self.sentbytescount = 0
40033 5e5b06087ec5 keepalive: track number of bytes received from an HTTP response Gregory Szorc <gregory.szorc@gmail.com> parents: 40032 diff changeset	702 self.receivedbytescount = 0
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	703
49543 76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	704 def __repr__(self):
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	705 base = super(HTTPConnection, self).__repr__()
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	706 local = "(unconnected)"
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	707 s = self.sock
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	708 if s:
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	709 try:
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	710 local = "%s:%d" % s.getsockname()
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	711 except OSError:
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	712 pass # Likely not connected
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	713 return "<%s: %s <--> %s:%d>" % (base, local, self.host, self.port)
76fbb1b6692a keepalive: add `__repr__()` to the HTTPConnection class to ease debugging Matt Harbison <matt_harbison@yahoo.com> parents: 49542 diff changeset	714
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	715
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	716 #########################################################################
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	717 ##### TEST FUNCTIONS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	718 #########################################################################
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	719
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	720
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	721 def continuity(url):
29341 0d83ad967bf8 cleanup: replace uses of util.(md5\|sha1\|sha256\|sha512) with hashlib.\1 Augie Fackler <raf@durin42.com> parents: 28883 diff changeset	722 md5 = hashlib.md5
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	723 format = b'%25s: %s'
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	724
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	725 # first fetch the file with the normal http handler
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	726 opener = urlreq.buildopener()
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	727 urlreq.installopener(opener)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	728 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	729 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	730 fo.close()
22505 232d437af120 keepalive: fix how md5 is used Mike Hommey <mh@glandium.org> parents: 19872 diff changeset	731 m = md5(foo)
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 45942 diff changeset	732 print(format % (b'normal urllib', hex(m.digest())))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	733
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	734 # now install the keepalive handler and try again
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	735 opener = urlreq.buildopener(HTTPHandler())
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	736 urlreq.installopener(opener)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	737
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	738 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	739 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	740 fo.close()
22505 232d437af120 keepalive: fix how md5 is used Mike Hommey <mh@glandium.org> parents: 19872 diff changeset	741 m = md5(foo)
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 45942 diff changeset	742 print(format % (b'keepalive read', hex(m.digest())))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	743
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	744 fo = urlreq.urlopen(url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	745 foo = b''
14494 1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression Martin Geisler <mg@lazybytes.net> parents: 10394 diff changeset	746 while True:
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	747 f = fo.readline()
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	748 if f:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	749 foo = foo + f
34435 5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34427 diff changeset	750 else:
5326e4ef1dab style: never put multiple statements on one line Alex Gaynor <agaynor@mozilla.com> parents: 34427 diff changeset	751 break
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	752 fo.close()
22505 232d437af120 keepalive: fix how md5 is used Mike Hommey <mh@glandium.org> parents: 19872 diff changeset	753 m = md5(foo)
46113 59fa3890d40a node: import symbols explicitly Joerg Sonnenberger <joerg@bec.de> parents: 45942 diff changeset	754 print(format % (b'keepalive readline', hex(m.digest())))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	755
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	756
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	757 def comp(N, url):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	758 print(b' making %i connections to:\n %s' % (N, url))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	759
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	760 procutil.stdout.write(b' first using the normal urllib handlers')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	761 # first use normal opener
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	762 opener = urlreq.buildopener()
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	763 urlreq.installopener(opener)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	764 t1 = fetch(N, url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	765 print(b' TIME: %.3f s' % t1)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	766
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	767 procutil.stdout.write(b' now using the keepalive handler ')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	768 # now install the keepalive handler and try again
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	769 opener = urlreq.buildopener(HTTPHandler())
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	770 urlreq.installopener(opener)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	771 t2 = fetch(N, url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	772 print(b' TIME: %.3f s' % t2)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	773 print(b' improvement factor: %.2f' % (t1 / t2))
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	774
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	775
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	776 def fetch(N, url, delay=0):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	777 import time
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	778
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	779 lens = []
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	780 starttime = time.time()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	781 for i in range(N):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	782 if delay and i > 0:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	783 time.sleep(delay)
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	784 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	785 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	786 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	787 lens.append(len(foo))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	788 diff = time.time() - starttime
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	789
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	790 j = 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	791 for i in lens[1:]:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	792 j = j + 1
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	793 if not i == lens[0]:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	794 print(b"WARNING: inconsistent length on read %i: %i" % (j, i))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	795
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	796 return diff
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	797
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	798
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	799 def test_timeout(url):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	800 global DEBUG
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	801 dbbackup = DEBUG
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	802
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48924 diff changeset	803 class FakeLogger:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	804 def debug(self, msg, *args):
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	805 print(msg % args)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	806
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	807 info = warning = error = debug
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	808
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	809 DEBUG = FakeLogger()
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	810 print(b" fetching the file to establish a connection")
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	811 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	812 data1 = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	813 fo.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	814
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	815 i = 20
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	816 print(b" waiting %i seconds for the server to close the connection" % i)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	817 while i > 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	818 procutil.stdout.write(b'\r %2i' % i)
37119 d4a2e0d5d042 procutil: bulk-replace util.std* to point to new module Yuya Nishihara <yuya@tcha.org> parents: 36295 diff changeset	819 procutil.stdout.flush()
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	820 time.sleep(1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	821 i -= 1
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	822 procutil.stderr.write(b'\r')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	823
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	824 print(b" fetching the file a second time")
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	825 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	826 data2 = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	827 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	828
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	829 if data1 == data2:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	830 print(b' data are identical')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	831 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	832 print(b' ERROR: DATA DIFFER')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	833
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	834 DEBUG = dbbackup
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	835
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	836
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	837 def test(url, N=10):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	838 print(b"performing continuity test (making sure stuff isn't corrupted)")
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	839 continuity(url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	840 print(b'')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	841 print(b"performing speed comparison")
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	842 comp(N, url)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	843 print(b'')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	844 print(b"performing dropped-connection check")
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	845 test_timeout(url)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	846
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	847
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	848 if __name__ == '__main__':
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	849 import time
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 41441 diff changeset	850
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	851 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	852 N = int(sys.argv[1])
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	853 url = sys.argv[2]
16688 cfb6682961b8 cleanup: replace naked excepts with more specific ones Brodie Rao <brodie@sf.io> parents: 16687 diff changeset	854 except (IndexError, ValueError):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	855 print(b"%s <integer> <url>" % sys.argv[0])
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	856 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	857 test(url, N)

Mercurial > public > mercurial-scm > hg

annotate mercurial/keepalive.py @ 51528:f85f23f1479b