mercurial-scm/hg-stable: mercurial/keepalive.py annotate

annotate mercurial/keepalive.py @ 34324:e45ec589f962

discovery: avoid dropping remote heads hidden locally An extra post processing was added to recognize remote heads that are hidden locally as "common" instead of "unknown". However, this processing was removing such hidden heads from the remote heads sets. It had no impact because we used to pull phase information from all remote heads. This series will replace the phase pulling operation to a more efficient process but requires the unmodified pulled set information.

author	Boris Feld <boris.feld@octobus.net>
date	Wed, 20 Sep 2017 05:47:33 +0200
parents	9bd003052d55
children	531332502568

rev	line source
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	1 # This library is free software; you can redistribute it and/or
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	2 # modify it under the terms of the GNU Lesser General Public
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	3 # License as published by the Free Software Foundation; either
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	4 # version 2.1 of the License, or (at your option) any later version.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	5 #
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	6 # This library is distributed in the hope that it will be useful,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	7 # but WITHOUT ANY WARRANTY; without even the implied warranty of
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	9 # Lesser General Public License for more details.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	10 #
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	11 # You should have received a copy of the GNU Lesser General Public
15782 7de7630053cb Remove FSF mailing address from GPL headers Martin Geisler <mg@aragost.com> parents: 14958 diff changeset	12 # License along with this library; if not, see
7de7630053cb Remove FSF mailing address from GPL headers Martin Geisler <mg@aragost.com> parents: 14958 diff changeset	13 # <http://www.gnu.org/licenses/>.
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	14
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	15 # This file is part of urlgrabber, a high-level cross-protocol url-grabber
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	16 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	17
4026 8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	18 # Modified by Benoit Boissinot:
8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	19 # - fix for digest auth (inspired from urllib2.py @ Python v2.4)
6470 ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions Dirkjan Ochtman <dirkjan@ochtman.nl> parents: 6001 diff changeset	20 # Modified by Dirkjan Ochtman:
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions Dirkjan Ochtman <dirkjan@ochtman.nl> parents: 6001 diff changeset	21 # - import md5 function from a local util module
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	22 # Modified by Augie Fackler:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	23 # - add safesend method and use it to prevent broken pipe errors
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	24 # on large POST requests
4026 8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	25
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	26 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	27
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	28 >>> import urllib2
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	29 >>> from keepalive import HTTPHandler
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	30 >>> keepalive_handler = HTTPHandler()
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	31 >>> opener = urlreq.buildopener(keepalive_handler)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	32 >>> urlreq.installopener(opener)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	33 >>>
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	34 >>> fo = urlreq.urlopen('http://www.python.org')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	35
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	36 If a connection to a given host is requested, and all of the existing
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	37 connections are still in use, another connection will be opened. If
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	38 the handler tries to use an existing connection but it fails in some
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	39 way, it will be closed and removed from the pool.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	40
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	41 To remove the handler, simply re-run build_opener with no arguments, and
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	42 install that opener.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	43
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	44 You can explicitly close connections by using the close_connection()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	45 method of the returned file-like object (described below) or you can
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	46 use the handler methods:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	47
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	48 close_connection(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	49 close_all()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	50 open_connections()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	51
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	52 NOTE: using the close_connection and close_all methods of the handler
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	53 should be done with care when using multiple threads.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	54 * there is nothing that prevents another thread from creating new
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	55 connections immediately after connections are closed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	56 * no checks are done to prevent in-use connections from being closed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	57
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	58 >>> keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	59
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	60 EXTRA ATTRIBUTES AND METHODS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	61
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	62 Upon a status of 200, the object returned has a few additional
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	63 attributes and methods, which should not be used if you want to
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	64 remain consistent with the normal urllib2-returned objects:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	65
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	66 close_connection() - close the connection to the host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	67 readlines() - you know, readlines()
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	68 status - the return status (i.e. 404)
e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	69 reason - english translation of status (i.e. 'File not found')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	70
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	71 If you want the best of both worlds, use this inside an
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	72 AttributeError-catching try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	73
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	74 >>> try: status = fo.status
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	75 >>> except AttributeError: status = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	76
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	77 Unfortunately, these are ONLY there if status == 200, so it's not
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	78 easy to distinguish between non-200 responses. The reason is that
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	79 urllib2 tries to do clever things with error codes 301, 302, 401,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	80 and 407, and it wraps the object upon return.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	81 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	82
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	83 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	84
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	85 from __future__ import absolute_import, print_function
27507 a16489f9132d keepalive: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	86
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	87 import errno
29341 0d83ad967bf8 cleanup: replace uses of util.(md5\|sha1\|sha256\|sha512) with hashlib.\1 Augie Fackler <raf@durin42.com> parents: 28883 diff changeset	88 import hashlib
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	89 import socket
27507 a16489f9132d keepalive: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	90 import sys
29456 e61d384e3be4 keepalive: switch from thread to threading module Pulkit Goyal <7895pulkit@gmail.com> parents: 29455 diff changeset	91 import threading
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	92
34315 9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32688 diff changeset	93 from .i18n import _
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	94 from . import (
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	95 util,
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	96 )
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	97
29455 0c741fd6158a py3: conditionalize httplib import Pulkit Goyal <7895pulkit@gmail.com> parents: 29341 diff changeset	98 httplib = util.httplib
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	99 urlerr = util.urlerr
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	100 urlreq = util.urlreq
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	101
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	102 DEBUG = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	103
14764 a7d5816087a9 classes: fix class style problems found by b071cd58af50 Thomas Arendsen Hein <thomas@intevation.de> parents: 14494 diff changeset	104 class ConnectionManager(object):
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	105 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	106 The connection manager must be able to:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	107 * keep track of all existing
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	108 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	109 def __init__(self):
29456 e61d384e3be4 keepalive: switch from thread to threading module Pulkit Goyal <7895pulkit@gmail.com> parents: 29455 diff changeset	110 self._lock = threading.Lock()
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	111 self._hostmap = {} # map hosts to a list of connections
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	112 self._connmap = {} # map connections to host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	113 self._readymap = {} # map connection to ready state
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	114
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	115 def add(self, host, connection, ready):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	116 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	117 try:
16686 67964cda8701 cleanup: "not x in y" -> "x not in y" Brodie Rao <brodie@sf.io> parents: 15782 diff changeset	118 if host not in self._hostmap:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	119 self._hostmap[host] = []
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	120 self._hostmap[host].append(connection)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	121 self._connmap[connection] = host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	122 self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	123 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	124 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	125
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	126 def remove(self, connection):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	127 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	128 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	129 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	130 host = self._connmap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	131 except KeyError:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	132 pass
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	133 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	134 del self._connmap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	135 del self._readymap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	136 self._hostmap[host].remove(connection)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	137 if not self._hostmap[host]: del self._hostmap[host]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	138 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	139 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	140
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	141 def set_ready(self, connection, ready):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	142 try:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	143 self._readymap[connection] = ready
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	144 except KeyError:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	145 pass
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	146
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	147 def get_ready_conn(self, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	148 conn = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	149 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	150 try:
5915 d0576d065993 Prefer i in d over d.has_key(i) Christian Ebert <blacktrash@gmx.net> parents: 4026 diff changeset	151 if host in self._hostmap:
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	152 for c in self._hostmap[host]:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	153 if self._readymap[c]:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	154 self._readymap[c] = 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	155 conn = c
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	156 break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	157 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	158 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	159 return conn
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	160
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	161 def get_all(self, host=None):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	162 if host:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	163 return list(self._hostmap.get(host, []))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	164 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	165 return dict(self._hostmap)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	166
14764 a7d5816087a9 classes: fix class style problems found by b071cd58af50 Thomas Arendsen Hein <thomas@intevation.de> parents: 14494 diff changeset	167 class KeepAliveHandler(object):
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	168 def __init__(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	169 self._cm = ConnectionManager()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	170
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	171 #### Connection Management
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	172 def open_connections(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	173 """return a list of connected hosts and the number of connections
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	174 to each. [('foo.com:80', 2), ('bar.org', 1)]"""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	175 return [(host, len(li)) for (host, li) in self._cm.get_all().items()]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	176
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	177 def close_connection(self, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	178 """close connection(s) to <host>
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	179 host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	180 no error occurs if there is no connection to that host."""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	181 for h in self._cm.get_all(host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	182 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	183 h.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	184
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	185 def close_all(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	186 """close all open connections"""
7622 4dd7b28003d2 use dict.iteritems() rather than dict.items() Dirkjan Ochtman <dirkjan@ochtman.nl> parents: 6470 diff changeset	187 for host, conns in self._cm.get_all().iteritems():
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	188 for h in conns:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	189 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	190 h.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	191
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	192 def _request_closed(self, request, host, connection):
17251 98166640b356 help: fix some instances of 'the the' Mads Kiilerich <mads@kiilerich.com> parents: 16705 diff changeset	193 """tells us that this request is now closed and that the
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	194 connection is ready for another request"""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	195 self._cm.set_ready(connection, 1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	196
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	197 def _remove_connection(self, host, connection, close=0):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	198 if close:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	199 connection.close()
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	200 self._cm.remove(connection)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	201
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	202 #### Transaction Execution
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	203 def http_open(self, req):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	204 return self.do_open(HTTPConnection, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	205
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	206 def do_open(self, http_class, req):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	207 host = req.get_host()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	208 if not host:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	209 raise urlerr.urlerror('no host given')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	210
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	211 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	212 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	213 while h:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	214 r = self._reuse_connection(h, req, host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	215
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	216 # if this response is non-None, then it worked and we're
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	217 # done. Break out, skipping the else block.
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	218 if r:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	219 break
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	220
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	221 # connection is bad - possibly closed by server
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	222 # discard it and ask for the next free connection
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	223 h.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	224 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	225 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	226 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	227 # no (working) free connections were found. Create a new one.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	228 h = http_class(host)
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	229 if DEBUG:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	230 DEBUG.info("creating new connection to %s (%d)",
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	231 host, id(h))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	232 self._cm.add(host, h, 0)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	233 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	234 r = h.getresponse()
34315 9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32688 diff changeset	235 # The string form of BadStatusLine is the status line. Add some context
9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32688 diff changeset	236 # to make the error message slightly more useful.
9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32688 diff changeset	237 except httplib.BadStatusLine as err:
9bd003052d55 keepalive: add more context to bad status line errors Gregory Szorc <gregory.szorc@gmail.com> parents: 32688 diff changeset	238 raise urlerr.urlerror(_('bad HTTP status line: %s') % err.line)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 22958 diff changeset	239 except (socket.error, httplib.HTTPException) as err:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	240 raise urlerr.urlerror(err)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	241
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	242 # if not a persistent connection, don't try to reuse it
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	243 if r.will_close:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	244 self._cm.remove(h)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	245
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	246 if DEBUG:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	247 DEBUG.info("STATUS: %s, %s", r.status, r.reason)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	248 r._handler = self
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	249 r._host = host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	250 r._url = req.get_full_url()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	251 r._connection = h
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	252 r.code = r.status
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	253 r.headers = r.msg
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	254 r.msg = r.reason
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	255
30496 88a448a12ae8 keepalive: discard legacy Python support for error handling Augie Fackler <augie@google.com> parents: 30482 diff changeset	256 return r
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	257
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	258 def _reuse_connection(self, h, req, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	259 """start the transaction with a re-used connection
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	260 return a response object (r) upon success or None on failure.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	261 This DOES not close or remove bad connections in cases where
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	262 it returns. However, if an unexpected exception occurs, it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	263 will close and remove the connection before re-raising.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	264 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	265 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	266 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	267 r = h.getresponse()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	268 # note: just because we got something back doesn't mean it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	269 # worked. We'll check the version below, too.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	270 except (socket.error, httplib.HTTPException):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	271 r = None
16705 c2d9ef43ff6c check-code: ignore naked excepts with a "re-raise" comment Brodie Rao <brodie@sf.io> parents: 16688 diff changeset	272 except: # re-raises
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	273 # adding this block just in case we've missed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	274 # something we will still raise the exception, but
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	275 # lets try and close the connection and remove it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	276 # first. We previously got into a nasty loop
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	277 # where an exception was uncaught, and so the
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	278 # connection stayed open. On the next try, the
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	279 # same exception was raised, etc. The trade-off is
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	280 # that it's now possible this call will raise
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	281 # a DIFFERENT exception
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	282 if DEBUG:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	283 DEBUG.error("unexpected exception - closing "
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	284 "connection to %s (%d)", host, id(h))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	285 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	286 h.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	287 raise
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	288
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	289 if r is None or r.version == 9:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	290 # httplib falls back to assuming HTTP 0.9 if it gets a
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	291 # bad header back. This is most likely to happen if
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	292 # the socket has been closed by the server since we
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	293 # last used the connection.
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	294 if DEBUG:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	295 DEBUG.info("failed to re-use connection to %s (%d)",
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	296 host, id(h))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	297 r = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	298 else:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	299 if DEBUG:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	300 DEBUG.info("re-using connection to %s (%d)", host, id(h))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	301
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	302 return r
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	303
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	304 def _start_transaction(self, h, req):
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	305 # What follows mostly reimplements HTTPConnection.request()
31999 aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	306 # except it adds self.parent.addheaders in the mix and sends headers
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	307 # in a deterministic order (to make testing easier).
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	308 headers = util.sortdict(self.parent.addheaders)
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	309 headers.update(sorted(req.headers.items()))
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	310 headers.update(sorted(req.unredirected_hdrs.items()))
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order Gregory Szorc <gregory.szorc@gmail.com> parents: 30942 diff changeset	311 headers = util.sortdict((n.lower(), v) for n, v in headers.items())
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	312 skipheaders = {}
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	313 for n in ('host', 'accept-encoding'):
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	314 if n in headers:
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	315 skipheaders['skip_' + n.replace('-', '_')] = 1
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	316 try:
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	317 if req.has_data():
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	318 data = req.get_data()
30942 1beeb5185930 keepalive: honor urllib2 style get_method overrides John Mulligan <phlogistonjohn@asynchrono.us> parents: 30688 diff changeset	319 h.putrequest(
1beeb5185930 keepalive: honor urllib2 style get_method overrides John Mulligan <phlogistonjohn@asynchrono.us> parents: 30688 diff changeset	320 req.get_method(), req.get_selector(), **skipheaders)
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	321 if 'content-type' not in headers:
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	322 h.putheader('Content-type',
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	323 'application/x-www-form-urlencoded')
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	324 if 'content-length' not in headers:
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	325 h.putheader('Content-length', '%d' % len(data))
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	326 else:
30942 1beeb5185930 keepalive: honor urllib2 style get_method overrides John Mulligan <phlogistonjohn@asynchrono.us> parents: 30688 diff changeset	327 h.putrequest(
1beeb5185930 keepalive: honor urllib2 style get_method overrides John Mulligan <phlogistonjohn@asynchrono.us> parents: 30688 diff changeset	328 req.get_method(), req.get_selector(), **skipheaders)
28278 b1b22185c764 keepalive: remove useless parentheses around exception type Gregory Szorc <gregory.szorc@gmail.com> parents: 27616 diff changeset	329 except socket.error as err:
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	330 raise urlerr.urlerror(err)
8233 655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs Patrick Mezard <pmezard@gmail.com> parents: 8146 diff changeset	331 for k, v in headers.items():
8146 4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	332 h.putheader(k, v)
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	333 h.endheaders()
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	334 if req.has_data():
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	335 h.send(data)
4f13ed6ee544 keepalive: attempt to fix issue1003 Matt Mackall <mpm@selenic.com> parents: 7875 diff changeset	336
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	337 class HTTPHandler(KeepAliveHandler, urlreq.httphandler):
5983 6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4026 diff changeset	338 pass
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 4026 diff changeset	339
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	340 class HTTPResponse(httplib.HTTPResponse):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	341 # we need to subclass HTTPResponse in order to
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	342 # 1) add readline() and readlines() methods
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	343 # 2) add close_connection() methods
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	344 # 3) add info() and geturl() methods
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	345
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	346 # in order to add readline(), read must be modified to deal with a
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	347 # buffer. example: readline must read a buffer and then spit back
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	348 # one line at a time. The only real alternative is to read one
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	349 # BYTE at a time (ick). Once something has been read, it can't be
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	350 # put back (ok, maybe it can, but that's even uglier than this),
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	351 # so if you THEN do a normal read, you must first take stuff from
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	352 # the buffer.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	353
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	354 # the read method wraps the original to accommodate buffering,
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	355 # although read() never adds to the buffer.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	356 # Both readline and readlines have been stolen with almost no
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	357 # modification from socket.py
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	358
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	359
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	360 def __init__(self, sock, debuglevel=0, strict=0, method=None):
32687 2806c7bbcb5e keepalive: pass the correct arguments to HTTPResponse Kyle Lippincott <spectral@google.com> parents: 31999 diff changeset	361 httplib.HTTPResponse.__init__(self, sock, debuglevel=debuglevel,
32688 d7626e85dada keepalive: set buffering=True to do more efficient reads of headers Kyle Lippincott <spectral@google.com> parents: 32687 diff changeset	362 strict=True, method=method,
d7626e85dada keepalive: set buffering=True to do more efficient reads of headers Kyle Lippincott <spectral@google.com> parents: 32687 diff changeset	363 buffering=True)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	364 self.fileno = sock.fileno
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	365 self.code = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	366 self._rbuf = ''
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	367 self._rbufsize = 8096
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	368 self._handler = None # inserted by the handler later
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	369 self._host = None # (same)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	370 self._url = None # (same)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	371 self._connection = None # (same)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	372
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	373 _raw_read = httplib.HTTPResponse.read
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	374
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	375 def close(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	376 if self.fp:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	377 self.fp.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	378 self.fp = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	379 if self._handler:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	380 self._handler._request_closed(self, self._host,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	381 self._connection)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	382
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	383 def close_connection(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	384 self._handler._remove_connection(self._host, self._connection, close=1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	385 self.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	386
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	387 def info(self):
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	388 return self.headers
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	389
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	390 def geturl(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	391 return self._url
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	392
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	393 def read(self, amt=None):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	394 # the _rbuf test is only in this first if for speed. It's not
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	395 # logically necessary
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	396 if self._rbuf and not amt is None:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	397 L = len(self._rbuf)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	398 if amt > L:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	399 amt -= L
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	400 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	401 s = self._rbuf[:amt]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	402 self._rbuf = self._rbuf[amt:]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	403 return s
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	404
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	405 s = self._rbuf + self._raw_read(amt)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	406 self._rbuf = ''
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	407 return s
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	408
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	409 # stolen from Python SVN #68532 to fix issue1088
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	410 def _read_chunked(self, amt):
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	411 chunk_left = self.chunk_left
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30496 diff changeset	412 parts = []
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	413
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	414 while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	415 if chunk_left is None:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	416 line = self.fp.readline()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	417 i = line.find(';')
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	418 if i >= 0:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	419 line = line[:i] # strip chunk-extensions
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	420 try:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	421 chunk_left = int(line, 16)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	422 except ValueError:
17424 e7cfe3587ea4 fix trivial spelling errors Mads Kiilerich <mads@kiilerich.com> parents: 17251 diff changeset	423 # close the connection as protocol synchronization is
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	424 # probably lost
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	425 self.close()
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30496 diff changeset	426 raise httplib.IncompleteRead(''.join(parts))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	427 if chunk_left == 0:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	428 break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	429 if amt is None:
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30496 diff changeset	430 parts.append(self._safe_read(chunk_left))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	431 elif amt < chunk_left:
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30496 diff changeset	432 parts.append(self._safe_read(amt))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	433 self.chunk_left = chunk_left - amt
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30496 diff changeset	434 return ''.join(parts)
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	435 elif amt == chunk_left:
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30496 diff changeset	436 parts.append(self._safe_read(amt))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	437 self._safe_read(2) # toss the CRLF at the end of the chunk
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	438 self.chunk_left = None
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30496 diff changeset	439 return ''.join(parts)
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	440 else:
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30496 diff changeset	441 parts.append(self._safe_read(chunk_left))
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	442 amt -= chunk_left
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	443
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	444 # we read the whole chunk, get another
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	445 self._safe_read(2) # toss the CRLF at the end of the chunk
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	446 chunk_left = None
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	447
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	448 # read and discard trailer up to the CRLF terminator
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	449 ### note: we shouldn't have any trailers!
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	450 while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	451 line = self.fp.readline()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	452 if not line:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	453 # a vanishingly small number of sites EOF without
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	454 # sending the trailer
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	455 break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	456 if line == '\r\n':
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	457 break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	458
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	459 # we read everything; close the "file"
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	460 self.close()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	461
30686 8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer Gregory Szorc <gregory.szorc@gmail.com> parents: 30496 diff changeset	462 return ''.join(parts)
7781 a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088) Matt Mackall <mpm@selenic.com> parents: 7622 diff changeset	463
30687 5d06f6b73a57 keepalive: remove limit argument from readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30686 diff changeset	464 def readline(self):
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	465 # Fast path for a line is already available in read buffer.
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	466 i = self._rbuf.find('\n')
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	467 if i >= 0:
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	468 i += 1
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	469 line = self._rbuf[:i]
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	470 self._rbuf = self._rbuf[i:]
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	471 return line
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	472
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	473 # No newline in local buffer. Read until we find one.
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	474 chunks = [self._rbuf]
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	475 i = -1
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	476 readsize = self._rbufsize
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	477 while True:
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	478 new = self._raw_read(readsize)
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	479 if not new:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	480 break
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	481
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	482 chunks.append(new)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	483 i = new.find('\n')
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	484 if i >= 0:
30688 dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	485 break
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	486
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	487 # We either have exhausted the stream or have a newline in chunks[-1].
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	488
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	489 # EOF
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	490 if i == -1:
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	491 self._rbuf = ''
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	492 return ''.join(chunks)
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	493
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	494 i += 1
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	495 self._rbuf = chunks[-1][i:]
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	496 chunks[-1] = chunks[-1][:i]
dc5b594f41e9 keepalive: rewrite readline() Gregory Szorc <gregory.szorc@gmail.com> parents: 30687 diff changeset	497 return ''.join(chunks)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	498
19872 681f7b9213a4 check-code: check for spaces around = for named parameters Mads Kiilerich <madski@unity3d.com> parents: 17700 diff changeset	499 def readlines(self, sizehint=0):
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	500 total = 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	501 list = []
14494 1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression Martin Geisler <mg@lazybytes.net> parents: 10394 diff changeset	502 while True:
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	503 line = self.readline()
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	504 if not line:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	505 break
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	506 list.append(line)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	507 total += len(line)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	508 if sizehint and total >= sizehint:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	509 break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	510 return list
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	511
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	512 def safesend(self, str):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	513 """Send `str' to the server.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	514
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	515 Shamelessly ripped off from httplib to patch a bad behavior.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	516 """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	517 # _broken_pipe_resp is an attribute we set in this function
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	518 # if the socket is closed while we're sending data but
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	519 # the server sent us a response before hanging up.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	520 # In that case, we want to pretend to send the rest of the
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	521 # outgoing data, and then let the user use getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	522 # (which we wrap) to get this last response before
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	523 # opening a new socket.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	524 if getattr(self, '_broken_pipe_resp', None) is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	525 return
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	526
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	527 if self.sock is None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	528 if self.auto_open:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	529 self.connect()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	530 else:
16687 e34106fa0dc3 cleanup: "raise SomeException()" -> "raise SomeException" Brodie Rao <brodie@sf.io> parents: 16686 diff changeset	531 raise httplib.NotConnected
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	532
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	533 # send the data to the server. if we get a broken pipe, then close
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	534 # the socket. we want to reconnect when somebody tries to send again.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	535 #
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	536 # NOTE: we DO propagate the error, though, because we cannot simply
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	537 # ignore the error... the caller will know if they can retry.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	538 if self.debuglevel > 0:
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	539 print("send:", repr(str))
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	540 try:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	541 blocksize = 8192
14958 fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	542 read = getattr(str, 'read', None)
fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	543 if read is not None:
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	544 if self.debuglevel > 0:
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	545 print("sending a read()able")
14958 fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	546 data = read(blocksize)
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	547 while data:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	548 self.sock.sendall(data)
14958 fd246aefedd3 keepalive: use safehasattr instead of hasattr Augie Fackler <durin42@gmail.com> parents: 14764 diff changeset	549 data = read(blocksize)
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	550 else:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	551 self.sock.sendall(str)
25660 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 22958 diff changeset	552 except socket.error as v:
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	553 reraise = True
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	554 if v[0] == errno.EPIPE: # Broken pipe
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	555 if self._HTTPConnection__state == httplib._CS_REQ_SENT:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	556 self._broken_pipe_resp = None
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	557 self._broken_pipe_resp = self.getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	558 reraise = False
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	559 self.close()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	560 if reraise:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	561 raise
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	562
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	563 def wrapgetresponse(cls):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	564 """Wraps getresponse in cls with a broken-pipe sane version.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	565 """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	566 def safegetresponse(self):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	567 # In safesend() we might set the _broken_pipe_resp
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	568 # attribute, in which case the socket has already
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	569 # been closed and we just need to give them the response
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	570 # back. Otherwise, we use the normal response path.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	571 r = getattr(self, '_broken_pipe_resp', None)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	572 if r is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	573 return r
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	574 return cls.getresponse(self)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	575 safegetresponse.__doc__ = cls.getresponse.__doc__
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	576 return safegetresponse
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	577
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	578 class HTTPConnection(httplib.HTTPConnection):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	579 # use the modified response class
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	580 response_class = HTTPResponse
9726 430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	581 send = safesend
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	582 getresponse = wrapgetresponse(httplib.HTTPConnection)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs Augie Fackler <durin42@gmail.com> parents: 8296 diff changeset	583
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	584
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	585 #########################################################################
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	586 ##### TEST FUNCTIONS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	587 #########################################################################
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	588
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	589
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	590 def continuity(url):
29341 0d83ad967bf8 cleanup: replace uses of util.(md5\|sha1\|sha256\|sha512) with hashlib.\1 Augie Fackler <raf@durin42.com> parents: 28883 diff changeset	591 md5 = hashlib.md5
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	592 format = '%25s: %s'
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	593
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	594 # first fetch the file with the normal http handler
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	595 opener = urlreq.buildopener()
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	596 urlreq.installopener(opener)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	597 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	598 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	599 fo.close()
22505 232d437af120 keepalive: fix how md5 is used Mike Hommey <mh@glandium.org> parents: 19872 diff changeset	600 m = md5(foo)
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	601 print(format % ('normal urllib', m.hexdigest()))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	602
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	603 # now install the keepalive handler and try again
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	604 opener = urlreq.buildopener(HTTPHandler())
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	605 urlreq.installopener(opener)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	606
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	607 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	608 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	609 fo.close()
22505 232d437af120 keepalive: fix how md5 is used Mike Hommey <mh@glandium.org> parents: 19872 diff changeset	610 m = md5(foo)
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	611 print(format % ('keepalive read', m.hexdigest()))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	612
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	613 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	614 foo = ''
14494 1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression Martin Geisler <mg@lazybytes.net> parents: 10394 diff changeset	615 while True:
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	616 f = fo.readline()
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	617 if f:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	618 foo = foo + f
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	619 else: break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	620 fo.close()
22505 232d437af120 keepalive: fix how md5 is used Mike Hommey <mh@glandium.org> parents: 19872 diff changeset	621 m = md5(foo)
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	622 print(format % ('keepalive readline', m.hexdigest()))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	623
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	624 def comp(N, url):
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	625 print(' making %i connections to:\n %s' % (N, url))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	626
30482 39d13b8c101d py3: bulk replace sys.stdin/out/err by util's Yuya Nishihara <yuya@tcha.org> parents: 30472 diff changeset	627 util.stdout.write(' first using the normal urllib handlers')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	628 # first use normal opener
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	629 opener = urlreq.buildopener()
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	630 urlreq.installopener(opener)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	631 t1 = fetch(N, url)
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	632 print(' TIME: %.3f s' % t1)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	633
30482 39d13b8c101d py3: bulk replace sys.stdin/out/err by util's Yuya Nishihara <yuya@tcha.org> parents: 30472 diff changeset	634 util.stdout.write(' now using the keepalive handler ')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	635 # now install the keepalive handler and try again
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	636 opener = urlreq.buildopener(HTTPHandler())
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	637 urlreq.installopener(opener)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	638 t2 = fetch(N, url)
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	639 print(' TIME: %.3f s' % t2)
0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	640 print(' improvement factor: %.2f' % (t1 / t2))
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	641
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	642 def fetch(N, url, delay=0):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	643 import time
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	644 lens = []
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	645 starttime = time.time()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	646 for i in range(N):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	647 if delay and i > 0:
08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	648 time.sleep(delay)
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	649 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	650 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	651 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	652 lens.append(len(foo))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	653 diff = time.time() - starttime
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	654
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	655 j = 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	656 for i in lens[1:]:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	657 j = j + 1
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	658 if not i == lens[0]:
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	659 print("WARNING: inconsistent length on read %i: %i" % (j, i))
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	660
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	661 return diff
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	662
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	663 def test_timeout(url):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	664 global DEBUG
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	665 dbbackup = DEBUG
14764 a7d5816087a9 classes: fix class style problems found by b071cd58af50 Thomas Arendsen Hein <thomas@intevation.de> parents: 14494 diff changeset	666 class FakeLogger(object):
10282 08a0f04b56bd many, many trivial check-code fixups Matt Mackall <mpm@selenic.com> parents: 9726 diff changeset	667 def debug(self, msg, *args):
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	668 print(msg % args)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	669 info = warning = error = debug
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	670 DEBUG = FakeLogger()
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	671 print(" fetching the file to establish a connection")
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	672 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	673 data1 = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	674 fo.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	675
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	676 i = 20
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	677 print(" waiting %i seconds for the server to close the connection" % i)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	678 while i > 0:
30482 39d13b8c101d py3: bulk replace sys.stdin/out/err by util's Yuya Nishihara <yuya@tcha.org> parents: 30472 diff changeset	679 util.stdout.write('\r %2i' % i)
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's Yuya Nishihara <yuya@tcha.org> parents: 30472 diff changeset	680 util.stdout.flush()
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	681 time.sleep(1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	682 i -= 1
30482 39d13b8c101d py3: bulk replace sys.stdin/out/err by util's Yuya Nishihara <yuya@tcha.org> parents: 30472 diff changeset	683 util.stderr.write('\r')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	684
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	685 print(" fetching the file a second time")
28883 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 28278 diff changeset	686 fo = urlreq.urlopen(url)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	687 data2 = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	688 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	689
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	690 if data1 == data2:
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	691 print(' data are identical')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	692 else:
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	693 print(' ERROR: DATA DIFFER')
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	694
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	695 DEBUG = dbbackup
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	696
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	697
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	698 def test(url, N=10):
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	699 print("performing continuity test (making sure stuff isn't corrupted)")
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	700 continuity(url)
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	701 print('')
0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	702 print("performing speed comparison")
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	703 comp(N, url)
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	704 print('')
0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	705 print("performing dropped-connection check")
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	706 test_timeout(url)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	707
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	708 if __name__ == '__main__':
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	709 import time
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	710 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	711 N = int(sys.argv[1])
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	712 url = sys.argv[2]
16688 cfb6682961b8 cleanup: replace naked excepts with more specific ones Brodie Rao <brodie@sf.io> parents: 16687 diff changeset	713 except (IndexError, ValueError):
27616 0765d8423fbc keepalive: use print function Gregory Szorc <gregory.szorc@gmail.com> parents: 27507 diff changeset	714 print("%s <integer> <url>" % sys.argv[0])
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	715 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	716 test(url, N)

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/keepalive.py @ 34324:e45ec589f962