mercurial-scm/hg: mercurial/keepalive.py annotate

annotate mercurial/keepalive.py @ 5909:f45f7390c1c5

strip: calculate list of extra nodes to save and pass it to changegroupsubset When we remove revision N from the repository, all revisions >= N are affected: either it's a descendant from N and will also be removed, or it's not a descendant of N and will be renumbered. As a consequence, we have to (at least temporarily) remove all filelog and manifest revisions that have a linkrev >= N, readding some of them later. Unfortunately, it's possible to have a revlog with two revisions r1 and r2 such that r1 < r2, but linkrev(r1) > linkrev(r2). If we try to strip revision linkrev(r1) from the repository, we'll also lose revision r2 when we truncate this revlog. We already use changegroupsubset to create a temporary changegroup containing the revisions that have to be restored, but that function is unable to detect that we also wanted to save the r2 in the case above. So we manually calculate these extra nodes and pass it to changegroupsubset. This should fix issue764.

author	Alexis S. L. Carvalho <alexis@cecm.usp.br>
date	Sat, 19 Jan 2008 18:01:16 -0200
parents	8520a773a141
children	d0576d065993 6f1fcbc58efa

rev	line source
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	1 # This library is free software; you can redistribute it and/or
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	2 # modify it under the terms of the GNU Lesser General Public
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	3 # License as published by the Free Software Foundation; either
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	4 # version 2.1 of the License, or (at your option) any later version.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	5 #
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	6 # This library is distributed in the hope that it will be useful,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	7 # but WITHOUT ANY WARRANTY; without even the implied warranty of
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	9 # Lesser General Public License for more details.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	10 #
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	11 # You should have received a copy of the GNU Lesser General Public
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	12 # License along with this library; if not, write to the
c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	13 # Free Software Foundation, Inc.,
c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	14 # 59 Temple Place, Suite 330,
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	15 # Boston, MA 02111-1307 USA
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	16
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	17 # This file is part of urlgrabber, a high-level cross-protocol url-grabber
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	18 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	19
4026 8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	20 # Modified by Benoit Boissinot:
8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	21 # - fix for digest auth (inspired from urllib2.py @ Python v2.4)
8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	22
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	23 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	24
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	25 >>> import urllib2
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	26 >>> from keepalive import HTTPHandler
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	27 >>> keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	28 >>> opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	29 >>> urllib2.install_opener(opener)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	30 >>>
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	31 >>> fo = urllib2.urlopen('http://www.python.org')
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	32
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	33 If a connection to a given host is requested, and all of the existing
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	34 connections are still in use, another connection will be opened. If
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	35 the handler tries to use an existing connection but it fails in some
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	36 way, it will be closed and removed from the pool.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	37
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	38 To remove the handler, simply re-run build_opener with no arguments, and
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	39 install that opener.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	40
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	41 You can explicitly close connections by using the close_connection()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	42 method of the returned file-like object (described below) or you can
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	43 use the handler methods:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	44
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	45 close_connection(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	46 close_all()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	47 open_connections()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	48
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	49 NOTE: using the close_connection and close_all methods of the handler
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	50 should be done with care when using multiple threads.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	51 * there is nothing that prevents another thread from creating new
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	52 connections immediately after connections are closed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	53 * no checks are done to prevent in-use connections from being closed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	54
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	55 >>> keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	56
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	57 EXTRA ATTRIBUTES AND METHODS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	58
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	59 Upon a status of 200, the object returned has a few additional
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	60 attributes and methods, which should not be used if you want to
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	61 remain consistent with the normal urllib2-returned objects:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	62
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	63 close_connection() - close the connection to the host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	64 readlines() - you know, readlines()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	65 status - the return status (ie 404)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	66 reason - english translation of status (ie 'File not found')
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	67
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	68 If you want the best of both worlds, use this inside an
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	69 AttributeError-catching try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	70
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	71 >>> try: status = fo.status
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	72 >>> except AttributeError: status = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	73
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	74 Unfortunately, these are ONLY there if status == 200, so it's not
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	75 easy to distinguish between non-200 responses. The reason is that
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	76 urllib2 tries to do clever things with error codes 301, 302, 401,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	77 and 407, and it wraps the object upon return.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	78
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	79 For python versions earlier than 2.4, you can avoid this fancy error
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	80 handling by setting the module-level global HANDLE_ERRORS to zero.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	81 You see, prior to 2.4, it's the HTTP Handler's job to determine what
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	82 to handle specially, and what to just pass up. HANDLE_ERRORS == 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	83 means "pass everything up". In python 2.4, however, this job no
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	84 longer belongs to the HTTP Handler and is now done by a NEW handler,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	85 HTTPErrorProcessor. Here's the bottom line:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	86
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	87 python version < 2.4
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	88 HANDLE_ERRORS == 1 (default) pass up 200, treat the rest as
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	89 errors
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	90 HANDLE_ERRORS == 0 pass everything up, error processing is
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	91 left to the calling code
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	92 python version >= 2.4
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	93 HANDLE_ERRORS == 1 pass up 200, treat the rest as errors
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	94 HANDLE_ERRORS == 0 (default) pass everything up, let the
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	95 other handlers (specifically,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	96 HTTPErrorProcessor) decide what to do
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	97
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	98 In practice, setting the variable either way makes little difference
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	99 in python 2.4, so for the most consistent behavior across versions,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	100 you probably just want to use the defaults, which will give you
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	101 exceptions on errors.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	102
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	103 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	104
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	105 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	106
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	107 import urllib2
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	108 import httplib
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	109 import socket
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	110 import thread
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	111
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	112 DEBUG = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	113
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	114 import sys
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	115 if sys.version_info < (2, 4): HANDLE_ERRORS = 1
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	116 else: HANDLE_ERRORS = 0
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	117
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	118 class ConnectionManager:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	119 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	120 The connection manager must be able to:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	121 * keep track of all existing
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	122 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	123 def __init__(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	124 self._lock = thread.allocate_lock()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	125 self._hostmap = {} # map hosts to a list of connections
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	126 self._connmap = {} # map connections to host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	127 self._readymap = {} # map connection to ready state
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	128
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	129 def add(self, host, connection, ready):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	130 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	131 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	132 if not self._hostmap.has_key(host): self._hostmap[host] = []
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	133 self._hostmap[host].append(connection)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	134 self._connmap[connection] = host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	135 self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	136 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	137 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	138
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	139 def remove(self, connection):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	140 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	141 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	142 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	143 host = self._connmap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	144 except KeyError:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	145 pass
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	146 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	147 del self._connmap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	148 del self._readymap[connection]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	149 self._hostmap[host].remove(connection)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	150 if not self._hostmap[host]: del self._hostmap[host]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	151 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	152 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	153
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	154 def set_ready(self, connection, ready):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	155 try: self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	156 except KeyError: pass
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	157
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	158 def get_ready_conn(self, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	159 conn = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	160 self._lock.acquire()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	161 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	162 if self._hostmap.has_key(host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	163 for c in self._hostmap[host]:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	164 if self._readymap[c]:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	165 self._readymap[c] = 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	166 conn = c
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	167 break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	168 finally:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	169 self._lock.release()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	170 return conn
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	171
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	172 def get_all(self, host=None):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	173 if host:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	174 return list(self._hostmap.get(host, []))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	175 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	176 return dict(self._hostmap)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	177
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	178 class HTTPHandler(urllib2.HTTPHandler):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	179 def __init__(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	180 self._cm = ConnectionManager()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	181
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	182 #### Connection Management
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	183 def open_connections(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	184 """return a list of connected hosts and the number of connections
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	185 to each. [('foo.com:80', 2), ('bar.org', 1)]"""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	186 return [(host, len(li)) for (host, li) in self._cm.get_all().items()]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	187
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	188 def close_connection(self, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	189 """close connection(s) to <host>
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	190 host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	191 no error occurs if there is no connection to that host."""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	192 for h in self._cm.get_all(host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	193 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	194 h.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	195
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	196 def close_all(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	197 """close all open connections"""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	198 for host, conns in self._cm.get_all().items():
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	199 for h in conns:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	200 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	201 h.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	202
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	203 def _request_closed(self, request, host, connection):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	204 """tells us that this request is now closed and the the
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	205 connection is ready for another request"""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	206 self._cm.set_ready(connection, 1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	207
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	208 def _remove_connection(self, host, connection, close=0):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	209 if close: connection.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	210 self._cm.remove(connection)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	211
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	212 #### Transaction Execution
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	213 def http_open(self, req):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	214 return self.do_open(HTTPConnection, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	215
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	216 def do_open(self, http_class, req):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	217 host = req.get_host()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	218 if not host:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	219 raise urllib2.URLError('no host given')
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	220
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	221 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	222 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	223 while h:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	224 r = self._reuse_connection(h, req, host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	225
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	226 # if this response is non-None, then it worked and we're
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	227 # done. Break out, skipping the else block.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	228 if r: break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	229
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	230 # connection is bad - possibly closed by server
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	231 # discard it and ask for the next free connection
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	232 h.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	233 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	234 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	235 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	236 # no (working) free connections were found. Create a new one.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	237 h = http_class(host)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	238 if DEBUG: DEBUG.info("creating new connection to %s (%d)",
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	239 host, id(h))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	240 self._cm.add(host, h, 0)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	241 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	242 r = h.getresponse()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	243 except (socket.error, httplib.HTTPException), err:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	244 raise urllib2.URLError(err)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	245
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	246 # if not a persistent connection, don't try to reuse it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	247 if r.will_close: self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	248
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	249 if DEBUG: DEBUG.info("STATUS: %s, %s", r.status, r.reason)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	250 r._handler = self
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	251 r._host = host
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	252 r._url = req.get_full_url()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	253 r._connection = h
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	254 r.code = r.status
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	255 r.headers = r.msg
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	256 r.msg = r.reason
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	257
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	258 if r.status == 200 or not HANDLE_ERRORS:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	259 return r
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	260 else:
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	261 return self.parent.error('http', req, r,
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	262 r.status, r.msg, r.headers)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	263
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	264 def _reuse_connection(self, h, req, host):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	265 """start the transaction with a re-used connection
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	266 return a response object (r) upon success or None on failure.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	267 This DOES not close or remove bad connections in cases where
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	268 it returns. However, if an unexpected exception occurs, it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	269 will close and remove the connection before re-raising.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	270 """
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	271 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	272 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	273 r = h.getresponse()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	274 # note: just because we got something back doesn't mean it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	275 # worked. We'll check the version below, too.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	276 except (socket.error, httplib.HTTPException):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	277 r = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	278 except:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	279 # adding this block just in case we've missed
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	280 # something we will still raise the exception, but
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	281 # lets try and close the connection and remove it
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	282 # first. We previously got into a nasty loop
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	283 # where an exception was uncaught, and so the
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	284 # connection stayed open. On the next try, the
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	285 # same exception was raised, etc. The tradeoff is
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	286 # that it's now possible this call will raise
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	287 # a DIFFERENT exception
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	288 if DEBUG: DEBUG.error("unexpected exception - closing " + \
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	289 "connection to %s (%d)", host, id(h))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	290 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	291 h.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	292 raise
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	293
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	294 if r is None or r.version == 9:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	295 # httplib falls back to assuming HTTP 0.9 if it gets a
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	296 # bad header back. This is most likely to happen if
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	297 # the socket has been closed by the server since we
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	298 # last used the connection.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	299 if DEBUG: DEBUG.info("failed to re-use connection to %s (%d)",
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	300 host, id(h))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	301 r = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	302 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	303 if DEBUG: DEBUG.info("re-using connection to %s (%d)", host, id(h))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	304
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	305 return r
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	306
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	307 def _start_transaction(self, h, req):
4026 8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	308 headers = req.headers.copy()
8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	309 body = req.data
8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	310 if sys.version_info >= (2, 4):
8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	311 headers.update(req.unredirected_hdrs)
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	312 try:
4026 8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	313 h.request(req.get_method(), req.get_selector(), body, headers)
8520a773a141 fix for digest auth when using keepalive.py Benoit Boissinot <benoit.boissinot@ens-lyon.org> parents: 2600 diff changeset	314 except socket.error, err: # XXX what error?
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	315 raise urllib2.URLError(err)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	316
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	317 class HTTPResponse(httplib.HTTPResponse):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	318 # we need to subclass HTTPResponse in order to
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	319 # 1) add readline() and readlines() methods
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	320 # 2) add close_connection() methods
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	321 # 3) add info() and geturl() methods
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	322
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	323 # in order to add readline(), read must be modified to deal with a
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	324 # buffer. example: readline must read a buffer and then spit back
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	325 # one line at a time. The only real alternative is to read one
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	326 # BYTE at a time (ick). Once something has been read, it can't be
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	327 # put back (ok, maybe it can, but that's even uglier than this),
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	328 # so if you THEN do a normal read, you must first take stuff from
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	329 # the buffer.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	330
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	331 # the read method wraps the original to accomodate buffering,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	332 # although read() never adds to the buffer.
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	333 # Both readline and readlines have been stolen with almost no
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	334 # modification from socket.py
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	335
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	336
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	337 def __init__(self, sock, debuglevel=0, strict=0, method=None):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	338 if method: # the httplib in python 2.3 uses the method arg
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	339 httplib.HTTPResponse.__init__(self, sock, debuglevel, method)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	340 else: # 2.2 doesn't
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	341 httplib.HTTPResponse.__init__(self, sock, debuglevel)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	342 self.fileno = sock.fileno
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	343 self.code = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	344 self._rbuf = ''
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	345 self._rbufsize = 8096
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	346 self._handler = None # inserted by the handler later
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	347 self._host = None # (same)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	348 self._url = None # (same)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	349 self._connection = None # (same)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	350
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	351 _raw_read = httplib.HTTPResponse.read
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	352
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	353 def close(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	354 if self.fp:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	355 self.fp.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	356 self.fp = None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	357 if self._handler:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	358 self._handler._request_closed(self, self._host,
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	359 self._connection)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	360
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	361 def close_connection(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	362 self._handler._remove_connection(self._host, self._connection, close=1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	363 self.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	364
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	365 def info(self):
2444 5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber. Alexis S. L. Carvalho <alexis@cecm.usp.br> parents: 2435 diff changeset	366 return self.headers
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	367
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	368 def geturl(self):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	369 return self._url
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	370
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	371 def read(self, amt=None):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	372 # the _rbuf test is only in this first if for speed. It's not
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	373 # logically necessary
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	374 if self._rbuf and not amt is None:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	375 L = len(self._rbuf)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	376 if amt > L:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	377 amt -= L
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	378 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	379 s = self._rbuf[:amt]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	380 self._rbuf = self._rbuf[amt:]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	381 return s
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	382
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	383 s = self._rbuf + self._raw_read(amt)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	384 self._rbuf = ''
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	385 return s
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	386
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	387 def readline(self, limit=-1):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	388 data = ""
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	389 i = self._rbuf.find('\n')
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	390 while i < 0 and not (0 < limit <= len(self._rbuf)):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	391 new = self._raw_read(self._rbufsize)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	392 if not new: break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	393 i = new.find('\n')
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	394 if i >= 0: i = i + len(self._rbuf)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	395 self._rbuf = self._rbuf + new
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	396 if i < 0: i = len(self._rbuf)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	397 else: i = i+1
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	398 if 0 <= limit < len(self._rbuf): i = limit
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	399 data, self._rbuf = self._rbuf[:i], self._rbuf[i:]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	400 return data
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	401
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	402 def readlines(self, sizehint = 0):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	403 total = 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	404 list = []
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	405 while 1:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	406 line = self.readline()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	407 if not line: break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	408 list.append(line)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	409 total += len(line)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	410 if sizehint and total >= sizehint:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	411 break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	412 return list
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	413
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	414
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	415 class HTTPConnection(httplib.HTTPConnection):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	416 # use the modified response class
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	417 response_class = HTTPResponse
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	418
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	419 #########################################################################
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	420 ##### TEST FUNCTIONS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	421 #########################################################################
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	422
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	423 def error_handler(url):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	424 global HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	425 orig = HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	426 keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	427 opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	428 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	429 pos = {0: 'off', 1: 'on'}
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	430 for i in (0, 1):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	431 print " fancy error handling %s (HANDLE_ERRORS = %i)" % (pos[i], i)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	432 HANDLE_ERRORS = i
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	433 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	434 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	435 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	436 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	437 try: status, reason = fo.status, fo.reason
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	438 except AttributeError: status, reason = None, None
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	439 except IOError, e:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	440 print " EXCEPTION: %s" % e
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	441 raise
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	442 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	443 print " status = %s, reason = %s" % (status, reason)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	444 HANDLE_ERRORS = orig
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	445 hosts = keepalive_handler.open_connections()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	446 print "open connections:", hosts
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	447 keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	448
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	449 def continuity(url):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	450 import md5
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	451 format = '%25s: %s'
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	452
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	453 # first fetch the file with the normal http handler
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	454 opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	455 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	456 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	457 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	458 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	459 m = md5.new(foo)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	460 print format % ('normal urllib', m.hexdigest())
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	461
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	462 # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	463 opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	464 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	465
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	466 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	467 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	468 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	469 m = md5.new(foo)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	470 print format % ('keepalive read', m.hexdigest())
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	471
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	472 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	473 foo = ''
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	474 while 1:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	475 f = fo.readline()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	476 if f: foo = foo + f
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	477 else: break
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	478 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	479 m = md5.new(foo)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	480 print format % ('keepalive readline', m.hexdigest())
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	481
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	482 def comp(N, url):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	483 print ' making %i connections to:\n %s' % (N, url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	484
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	485 sys.stdout.write(' first using the normal urllib handlers')
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	486 # first use normal opener
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	487 opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	488 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	489 t1 = fetch(N, url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	490 print ' TIME: %.3f s' % t1
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	491
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	492 sys.stdout.write(' now using the keepalive handler ')
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	493 # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	494 opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	495 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	496 t2 = fetch(N, url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	497 print ' TIME: %.3f s' % t2
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	498 print ' improvement factor: %.2f' % (t1/t2, )
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	499
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	500 def fetch(N, url, delay=0):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	501 import time
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	502 lens = []
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	503 starttime = time.time()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	504 for i in range(N):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	505 if delay and i > 0: time.sleep(delay)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	506 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	507 foo = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	508 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	509 lens.append(len(foo))
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	510 diff = time.time() - starttime
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	511
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	512 j = 0
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	513 for i in lens[1:]:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	514 j = j + 1
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	515 if not i == lens[0]:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	516 print "WARNING: inconsistent length on read %i: %i" % (j, i)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	517
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	518 return diff
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	519
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	520 def test_timeout(url):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	521 global DEBUG
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	522 dbbackup = DEBUG
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	523 class FakeLogger:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	524 def debug(self, msg, *args): print msg % args
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	525 info = warning = error = debug
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	526 DEBUG = FakeLogger()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	527 print " fetching the file to establish a connection"
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	528 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	529 data1 = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	530 fo.close()
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	531
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	532 i = 20
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	533 print " waiting %i seconds for the server to close the connection" % i
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	534 while i > 0:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	535 sys.stdout.write('\r %2i' % i)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	536 sys.stdout.flush()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	537 time.sleep(1)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	538 i -= 1
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	539 sys.stderr.write('\r')
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	540
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	541 print " fetching the file a second time"
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	542 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	543 data2 = fo.read()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	544 fo.close()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	545
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	546 if data1 == data2:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	547 print ' data are identical'
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	548 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	549 print ' ERROR: DATA DIFFER'
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	550
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	551 DEBUG = dbbackup
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	552
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	553
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	554 def test(url, N=10):
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	555 print "checking error hander (do this on a non-200)"
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	556 try: error_handler(url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	557 except IOError, e:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	558 print "exiting - exception will prevent further tests"
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	559 sys.exit()
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	560 print
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	561 print "performing continuity test (making sure stuff isn't corrupted)"
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	562 continuity(url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	563 print
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	564 print "performing speed comparison"
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	565 comp(N, url)
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	566 print
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	567 print "performing dropped-connection check"
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	568 test_timeout(url)
2600 c4325f0a9b91 clean up trailing white space. Vadim Gelfer <vadim.gelfer@gmail.com> parents: 2444 diff changeset	569
2435 ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	570 if __name__ == '__main__':
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	571 import time
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	572 import sys
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	573 try:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	574 N = int(sys.argv[1])
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	575 url = sys.argv[2]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	576 except:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	577 print "%s <integer> <url>" % sys.argv[0]
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	578 else:
ff2bac730b99 http client: support persistent connections. Vadim Gelfer <vadim.gelfer@gmail.com> parents: diff changeset	579 test(url, N)

Mercurial > public > mercurial-scm > hg

annotate mercurial/keepalive.py @ 5909:f45f7390c1c5