mercurial-scm/hg-stable: hgext/remotefilelog/fileserverclient.py annotate

annotate hgext/remotefilelog/fileserverclient.py @ 40865:b34b1b865ef9

remotefilelog: avoid temporarily using "count" variable as synonym for "total" The "count" variable is generally used for updating progress, but early in fileserverclient.request(), its used to mean the total count. We already have another "total" variable for that, so it seems much clearer to use that. Differential Revision: https://phab.mercurial-scm.org/D5382

author	Martin von Zweigbergk <martinvonz@google.com>
date	Tue, 04 Dec 2018 21:58:46 -0800
parents	e58cd7ede1c3
children	70de33b9d328

rev	line source
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	1 # fileserverclient.py - client for communicating with the cache process
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	2 #
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	3 # Copyright 2013 Facebook, Inc.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	4 #
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	7
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	8 from __future__ import absolute_import
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	9
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	10 import hashlib
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	11 import io
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	12 import os
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	13 import threading
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	14 import time
40557 e2a1584e9e3f remotefilelog: rip out lz4 support Augie Fackler <augie@google.com> parents: 40552 diff changeset	15 import zlib
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	16
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	17 from mercurial.i18n import _
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	18 from mercurial.node import bin, hex, nullid
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	19 from mercurial import (
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	20 error,
40659 3fa4183e7803 py3: use node.hex(h.digest()) instead of h.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40657 diff changeset	21 node,
40657 13d4ad8d7801 py3: fix keyword arguments handling in hgext/remotefilelog/ Pulkit Goyal <pulkit@yandex-team.ru> parents: 40561 diff changeset	22 pycompat,
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	23 revlog,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	24 sshpeer,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	25 util,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	26 wireprotov1peer,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	27 )
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	28 from mercurial.utils import procutil
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	29
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	30 from . import (
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	31 constants,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	32 contentstore,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	33 metadatastore,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	34 )
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	35
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	36 _sshv1peer = sshpeer.sshv1peer
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	37
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	38 # Statistics for debugging
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	39 fetchcost = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	40 fetches = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	41 fetched = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	42 fetchmisses = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	43
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	44 _lfsmod = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	45
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	46 def getcachekey(reponame, file, id):
40659 3fa4183e7803 py3: use node.hex(h.digest()) instead of h.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40657 diff changeset	47 pathhash = node.hex(hashlib.sha1(file).digest())
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	48 return os.path.join(reponame, pathhash[:2], pathhash[2:], id)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	49
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	50 def getlocalkey(file, id):
40659 3fa4183e7803 py3: use node.hex(h.digest()) instead of h.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40657 diff changeset	51 pathhash = node.hex(hashlib.sha1(file).digest())
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	52 return os.path.join(pathhash, id)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	53
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	54 def peersetup(ui, peer):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	55
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	56 class remotefilepeer(peer.__class__):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	57 @wireprotov1peer.batchable
40561 466dd4d70bff remotefilelog: rename wireproto methods and remaining capabilities Augie Fackler <augie@google.com> parents: 40560 diff changeset	58 def x_rfl_getfile(self, file, node):
466dd4d70bff remotefilelog: rename wireproto methods and remaining capabilities Augie Fackler <augie@google.com> parents: 40560 diff changeset	59 if not self.capable('x_rfl_getfile'):
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	60 raise error.Abort(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	61 'configured remotefile server does not support getfile')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	62 f = wireprotov1peer.future()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	63 yield {'file': file, 'node': node}, f
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	64 code, data = f.value.split('\0', 1)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	65 if int(code):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	66 raise error.LookupError(file, node, data)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	67 yield data
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	68
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	69 @wireprotov1peer.batchable
40561 466dd4d70bff remotefilelog: rename wireproto methods and remaining capabilities Augie Fackler <augie@google.com> parents: 40560 diff changeset	70 def x_rfl_getflogheads(self, path):
466dd4d70bff remotefilelog: rename wireproto methods and remaining capabilities Augie Fackler <augie@google.com> parents: 40560 diff changeset	71 if not self.capable('x_rfl_getflogheads'):
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	72 raise error.Abort('configured remotefile server does not '
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	73 'support getflogheads')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	74 f = wireprotov1peer.future()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	75 yield {'path': path}, f
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	76 heads = f.value.split('\n') if f.value else []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	77 yield heads
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	78
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	79 def _updatecallstreamopts(self, command, opts):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	80 if command != 'getbundle':
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	81 return
40558 354acd0dc637 remotefilelog: rename capability for legacy ssh file fetching method Augie Fackler <augie@google.com> parents: 40557 diff changeset	82 if (constants.NETWORK_CAP_LEGACY_SSH_GETFILES
354acd0dc637 remotefilelog: rename capability for legacy ssh file fetching method Augie Fackler <augie@google.com> parents: 40557 diff changeset	83 not in self.capabilities()):
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	84 return
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	85 if not util.safehasattr(self, '_localrepo'):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	86 return
40560 fc2766860796 remotefilelog: consolidate and rename on-disk store requirement Augie Fackler <augie@google.com> parents: 40559 diff changeset	87 if (constants.SHALLOWREPO_REQUIREMENT
fc2766860796 remotefilelog: consolidate and rename on-disk store requirement Augie Fackler <augie@google.com> parents: 40559 diff changeset	88 not in self._localrepo.requirements):
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	89 return
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	90
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	91 bundlecaps = opts.get('bundlecaps')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	92 if bundlecaps:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	93 bundlecaps = [bundlecaps]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	94 else:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	95 bundlecaps = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	96
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	97 # shallow, includepattern, and excludepattern are a hacky way of
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	98 # carrying over data from the local repo to this getbundle
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	99 # command. We need to do it this way because bundle1 getbundle
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	100 # doesn't provide any other place we can hook in to manipulate
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	101 # getbundle args before it goes across the wire. Once we get rid
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	102 # of bundle1, we can use bundle2's _pullbundle2extraprepare to
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	103 # do this more cleanly.
40559 ed19958dbf5d remotefilelog: consolidate and rename bundle2 capability Augie Fackler <augie@google.com> parents: 40558 diff changeset	104 bundlecaps.append(constants.BUNDLE2_CAPABLITY)
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	105 if self._localrepo.includepattern:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	106 patterns = '\0'.join(self._localrepo.includepattern)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	107 includecap = "includepattern=" + patterns
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	108 bundlecaps.append(includecap)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	109 if self._localrepo.excludepattern:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	110 patterns = '\0'.join(self._localrepo.excludepattern)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	111 excludecap = "excludepattern=" + patterns
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	112 bundlecaps.append(excludecap)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	113 opts['bundlecaps'] = ','.join(bundlecaps)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	114
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	115 def _sendrequest(self, command, args, **opts):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	116 self._updatecallstreamopts(command, args)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	117 return super(remotefilepeer, self)._sendrequest(command, args,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	118 **opts)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	119
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	120 def _callstream(self, command, **opts):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	121 supertype = super(remotefilepeer, self)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	122 if not util.safehasattr(supertype, '_sendrequest'):
40657 13d4ad8d7801 py3: fix keyword arguments handling in hgext/remotefilelog/ Pulkit Goyal <pulkit@yandex-team.ru> parents: 40561 diff changeset	123 self._updatecallstreamopts(command, pycompat.byteskwargs(opts))
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	124 return super(remotefilepeer, self)._callstream(command, **opts)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	125
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	126 peer.__class__ = remotefilepeer
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	127
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	128 class cacheconnection(object):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	129 """The connection for communicating with the remote cache. Performs
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	130 gets and sets by communicating with an external process that has the
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	131 cache-specific implementation.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	132 """
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	133 def __init__(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	134 self.pipeo = self.pipei = self.pipee = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	135 self.subprocess = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	136 self.connected = False
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	137
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	138 def connect(self, cachecommand):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	139 if self.pipeo:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	140 raise error.Abort(_("cache connection already open"))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	141 self.pipei, self.pipeo, self.pipee, self.subprocess = \
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	142 procutil.popen4(cachecommand)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	143 self.connected = True
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	144
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	145 def close(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	146 def tryclose(pipe):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	147 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	148 pipe.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	149 except Exception:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	150 pass
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	151 if self.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	152 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	153 self.pipei.write("exit\n")
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	154 except Exception:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	155 pass
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	156 tryclose(self.pipei)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	157 self.pipei = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	158 tryclose(self.pipeo)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	159 self.pipeo = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	160 tryclose(self.pipee)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	161 self.pipee = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	162 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	163 # Wait for process to terminate, making sure to avoid deadlock.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	164 # See https://docs.python.org/2/library/subprocess.html for
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	165 # warnings about wait() and deadlocking.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	166 self.subprocess.communicate()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	167 except Exception:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	168 pass
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	169 self.subprocess = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	170 self.connected = False
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	171
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	172 def request(self, request, flush=True):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	173 if self.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	174 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	175 self.pipei.write(request)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	176 if flush:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	177 self.pipei.flush()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	178 except IOError:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	179 self.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	180
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	181 def receiveline(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	182 if not self.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	183 return None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	184 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	185 result = self.pipeo.readline()[:-1]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	186 if not result:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	187 self.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	188 except IOError:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	189 self.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	190
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	191 return result
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	192
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	193 def _getfilesbatch(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	194 remote, receivemissing, progresstick, missed, idmap, batchsize):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	195 # Over http(s), iterbatch is a streamy method and we can start
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	196 # looking at results early. This means we send one (potentially
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	197 # large) request, but then we show nice progress as we process
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	198 # file results, rather than showing chunks of $batchsize in
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	199 # progress.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	200 #
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	201 # Over ssh, iterbatch isn't streamy because batch() wasn't
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	202 # explicitly designed as a streaming method. In the future we
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	203 # should probably introduce a streambatch() method upstream and
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	204 # use that for this.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	205 with remote.commandexecutor() as e:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	206 futures = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	207 for m in missed:
40561 466dd4d70bff remotefilelog: rename wireproto methods and remaining capabilities Augie Fackler <augie@google.com> parents: 40560 diff changeset	208 futures.append(e.callcommand('x_rfl_getfile', {
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	209 'file': idmap[m],
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	210 'node': m[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	211 }))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	212
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	213 for i, m in enumerate(missed):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	214 r = futures[i].result()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	215 futures[i] = None # release memory
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	216 file_ = idmap[m]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	217 node = m[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	218 receivemissing(io.BytesIO('%d\n%s' % (len(r), r)), file_, node)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	219 progresstick()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	220
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	221 def _getfiles_optimistic(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	222 remote, receivemissing, progresstick, missed, idmap, step):
40561 466dd4d70bff remotefilelog: rename wireproto methods and remaining capabilities Augie Fackler <augie@google.com> parents: 40560 diff changeset	223 remote._callstream("x_rfl_getfiles")
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	224 i = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	225 pipeo = remote._pipeo
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	226 pipei = remote._pipei
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	227 while i < len(missed):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	228 # issue a batch of requests
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	229 start = i
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	230 end = min(len(missed), start + step)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	231 i = end
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	232 for missingid in missed[start:end]:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	233 # issue new request
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	234 versionid = missingid[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	235 file = idmap[missingid]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	236 sshrequest = "%s%s\n" % (versionid, file)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	237 pipeo.write(sshrequest)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	238 pipeo.flush()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	239
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	240 # receive batch results
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	241 for missingid in missed[start:end]:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	242 versionid = missingid[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	243 file = idmap[missingid]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	244 receivemissing(pipei, file, versionid)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	245 progresstick()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	246
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	247 # End the command
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	248 pipeo.write('\n')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	249 pipeo.flush()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	250
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	251 def _getfiles_threaded(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	252 remote, receivemissing, progresstick, missed, idmap, step):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	253 remote._callstream("getfiles")
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	254 pipeo = remote._pipeo
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	255 pipei = remote._pipei
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	256
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	257 def writer():
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	258 for missingid in missed:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	259 versionid = missingid[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	260 file = idmap[missingid]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	261 sshrequest = "%s%s\n" % (versionid, file)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	262 pipeo.write(sshrequest)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	263 pipeo.flush()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	264 writerthread = threading.Thread(target=writer)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	265 writerthread.daemon = True
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	266 writerthread.start()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	267
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	268 for missingid in missed:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	269 versionid = missingid[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	270 file = idmap[missingid]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	271 receivemissing(pipei, file, versionid)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	272 progresstick()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	273
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	274 writerthread.join()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	275 # End the command
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	276 pipeo.write('\n')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	277 pipeo.flush()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	278
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	279 class fileserverclient(object):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	280 """A client for requesting files from the remote file server.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	281 """
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	282 def __init__(self, repo):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	283 ui = repo.ui
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	284 self.repo = repo
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	285 self.ui = ui
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	286 self.cacheprocess = ui.config("remotefilelog", "cacheprocess")
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	287 if self.cacheprocess:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	288 self.cacheprocess = util.expandpath(self.cacheprocess)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	289
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	290 # This option causes remotefilelog to pass the full file path to the
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	291 # cacheprocess instead of a hashed key.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	292 self.cacheprocesspasspath = ui.configbool(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	293 "remotefilelog", "cacheprocess.includepath")
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	294
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	295 self.debugoutput = ui.configbool("remotefilelog", "debug")
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	296
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	297 self.remotecache = cacheconnection()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	298
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	299 def setstore(self, datastore, historystore, writedata, writehistory):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	300 self.datastore = datastore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	301 self.historystore = historystore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	302 self.writedata = writedata
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	303 self.writehistory = writehistory
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	304
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	305 def _connect(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	306 return self.repo.connectionpool.get(self.repo.fallbackpath)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	307
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	308 def request(self, fileids):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	309 """Takes a list of filename/node pairs and fetches them from the
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	310 server. Files are stored in the local cache.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	311 A list of nodes that the server couldn't find is returned.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	312 If the connection fails, an exception is raised.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	313 """
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	314 if not self.remotecache.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	315 self.connect()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	316 cache = self.remotecache
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	317 writedata = self.writedata
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	318
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	319 repo = self.repo
40865 b34b1b865ef9 remotefilelog: avoid temporarily using "count" variable as synonym for "total" Martin von Zweigbergk <martinvonz@google.com> parents: 40864 diff changeset	320 total = len(fileids)
b34b1b865ef9 remotefilelog: avoid temporarily using "count" variable as synonym for "total" Martin von Zweigbergk <martinvonz@google.com> parents: 40864 diff changeset	321 request = "get\n%d\n" % total
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	322 idmap = {}
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	323 reponame = repo.name
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	324 for file, id in fileids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	325 fullid = getcachekey(reponame, file, id)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	326 if self.cacheprocesspasspath:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	327 request += file + '\0'
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	328 request += fullid + "\n"
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	329 idmap[fullid] = file
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	330
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	331 cache.request(request)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	332
40865 b34b1b865ef9 remotefilelog: avoid temporarily using "count" variable as synonym for "total" Martin von Zweigbergk <martinvonz@google.com> parents: 40864 diff changeset	333 progress = self.ui.makeprogress(_('downloading'), total=total)
40864 e58cd7ede1c3 remotefilelog: use progress helper in fileserverclient Martin von Zweigbergk <martinvonz@google.com> parents: 40661 diff changeset	334 progress.update(0)
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	335
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	336 missed = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	337 count = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	338 while True:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	339 missingid = cache.receiveline()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	340 if not missingid:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	341 missedset = set(missed)
40660 9769e0f6ffe0 py3: don't use dict.iterkeys() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40659 diff changeset	342 for missingid in idmap:
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	343 if not missingid in missedset:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	344 missed.append(missingid)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	345 self.ui.warn(_("warning: cache connection closed early - " +
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	346 "falling back to server\n"))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	347 break
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	348 if missingid == "0":
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	349 break
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	350 if missingid.startswith("_hits_"):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	351 # receive progress reports
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	352 parts = missingid.split("_")
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	353 count += int(parts[2])
40864 e58cd7ede1c3 remotefilelog: use progress helper in fileserverclient Martin von Zweigbergk <martinvonz@google.com> parents: 40661 diff changeset	354 progress.update(count)
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	355 continue
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	356
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	357 missed.append(missingid)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	358
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	359 global fetchmisses
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	360 fetchmisses += len(missed)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	361
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	362 count = [total - len(missed)]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	363 fromcache = count[0]
40864 e58cd7ede1c3 remotefilelog: use progress helper in fileserverclient Martin von Zweigbergk <martinvonz@google.com> parents: 40661 diff changeset	364 progress.update(count[0], total=total)
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	365 self.ui.log("remotefilelog", "remote cache hit rate is %r of %r\n",
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	366 count[0], total, hit=count[0], total=total)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	367
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	368 oldumask = os.umask(0o002)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	369 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	370 # receive cache misses from master
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	371 if missed:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	372 def progresstick():
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	373 count[0] += 1
40864 e58cd7ede1c3 remotefilelog: use progress helper in fileserverclient Martin von Zweigbergk <martinvonz@google.com> parents: 40661 diff changeset	374 progress.update(count[0])
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	375 # When verbose is true, sshpeer prints 'running ssh...'
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	376 # to stdout, which can interfere with some command
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	377 # outputs
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	378 verbose = self.ui.verbose
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	379 self.ui.verbose = False
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	380 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	381 with self._connect() as conn:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	382 remote = conn.peer
40558 354acd0dc637 remotefilelog: rename capability for legacy ssh file fetching method Augie Fackler <augie@google.com> parents: 40557 diff changeset	383 if remote.capable(
354acd0dc637 remotefilelog: rename capability for legacy ssh file fetching method Augie Fackler <augie@google.com> parents: 40557 diff changeset	384 constants.NETWORK_CAP_LEGACY_SSH_GETFILES):
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	385 if not isinstance(remote, _sshv1peer):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	386 raise error.Abort('remotefilelog requires ssh '
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	387 'servers')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	388 step = self.ui.configint('remotefilelog',
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	389 'getfilesstep')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	390 getfilestype = self.ui.config('remotefilelog',
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	391 'getfilestype')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	392 if getfilestype == 'threaded':
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	393 _getfiles = _getfiles_threaded
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	394 else:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	395 _getfiles = _getfiles_optimistic
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	396 _getfiles(remote, self.receivemissing, progresstick,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	397 missed, idmap, step)
40561 466dd4d70bff remotefilelog: rename wireproto methods and remaining capabilities Augie Fackler <augie@google.com> parents: 40560 diff changeset	398 elif remote.capable("x_rfl_getfile"):
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	399 if remote.capable('batch'):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	400 batchdefault = 100
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	401 else:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	402 batchdefault = 10
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	403 batchsize = self.ui.configint(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	404 'remotefilelog', 'batchsize', batchdefault)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	405 _getfilesbatch(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	406 remote, self.receivemissing, progresstick,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	407 missed, idmap, batchsize)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	408 else:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	409 raise error.Abort("configured remotefilelog server"
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	410 " does not support remotefilelog")
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	411
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	412 self.ui.log("remotefilefetchlog",
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	413 "Success\n",
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	414 fetched_files = count[0] - fromcache,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	415 total_to_fetch = total - fromcache)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	416 except Exception:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	417 self.ui.log("remotefilefetchlog",
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	418 "Fail\n",
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	419 fetched_files = count[0] - fromcache,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	420 total_to_fetch = total - fromcache)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	421 raise
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	422 finally:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	423 self.ui.verbose = verbose
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	424 # send to memcache
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	425 count[0] = len(missed)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	426 request = "set\n%d\n%s\n" % (count[0], "\n".join(missed))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	427 cache.request(request)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	428
40864 e58cd7ede1c3 remotefilelog: use progress helper in fileserverclient Martin von Zweigbergk <martinvonz@google.com> parents: 40661 diff changeset	429 progress.complete()
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	430
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	431 # mark ourselves as a user of this cache
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	432 writedata.markrepo(self.repo.path)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	433 finally:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	434 os.umask(oldumask)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	435
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	436 def receivemissing(self, pipe, filename, node):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	437 line = pipe.readline()[:-1]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	438 if not line:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	439 raise error.ResponseError(_("error downloading file contents:"),
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	440 _("connection closed early"))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	441 size = int(line)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	442 data = pipe.read(size)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	443 if len(data) != size:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	444 raise error.ResponseError(_("error downloading file contents:"),
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	445 _("only received %s of %s bytes")
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	446 % (len(data), size))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	447
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	448 self.writedata.addremotefilelognode(filename, bin(node),
40557 e2a1584e9e3f remotefilelog: rip out lz4 support Augie Fackler <augie@google.com> parents: 40552 diff changeset	449 zlib.decompress(data))
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	450
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	451 def connect(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	452 if self.cacheprocess:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	453 cmd = "%s %s" % (self.cacheprocess, self.writedata._path)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	454 self.remotecache.connect(cmd)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	455 else:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	456 # If no cache process is specified, we fake one that always
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	457 # returns cache misses. This enables tests to run easily
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	458 # and may eventually allow us to be a drop in replacement
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	459 # for the largefiles extension.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	460 class simplecache(object):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	461 def __init__(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	462 self.missingids = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	463 self.connected = True
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	464
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	465 def close(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	466 pass
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	467
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	468 def request(self, value, flush=True):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	469 lines = value.split("\n")
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	470 if lines[0] != "get":
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	471 return
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	472 self.missingids = lines[2:-1]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	473 self.missingids.append('0')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	474
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	475 def receiveline(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	476 if len(self.missingids) > 0:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	477 return self.missingids.pop(0)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	478 return None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	479
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	480 self.remotecache = simplecache()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	481
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	482 def close(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	483 if fetches:
40661 aa588bf40a08 py3: add b suffix to make sure file is opened in bytes mode Pulkit Goyal <pulkit@yandex-team.ru> parents: 40660 diff changeset	484 msg = ("%d files fetched over %d fetches - " +
40545 3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	485 "(%d misses, %0.2f%% hit ratio) over %0.2fs\n") % (
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	486 fetched,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	487 fetches,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	488 fetchmisses,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	489 float(fetched - fetchmisses) / float(fetched) * 100.0,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	490 fetchcost)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	491 if self.debugoutput:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	492 self.ui.warn(msg)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	493 self.ui.log("remotefilelog.prefetch", msg.replace("%", "%%"),
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	494 remotefilelogfetched=fetched,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	495 remotefilelogfetches=fetches,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	496 remotefilelogfetchmisses=fetchmisses,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	497 remotefilelogfetchtime=fetchcost * 1000)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	498
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	499 if self.remotecache.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	500 self.remotecache.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	501
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	502 def prefetch(self, fileids, force=False, fetchdata=True,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	503 fetchhistory=False):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	504 """downloads the given file versions to the cache
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	505 """
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	506 repo = self.repo
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	507 idstocheck = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	508 for file, id in fileids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	509 # hack
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	510 # - we don't use .hgtags
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	511 # - workingctx produces ids with length 42,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	512 # which we skip since they aren't in any cache
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	513 if (file == '.hgtags' or len(id) == 42
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	514 or not repo.shallowmatch(file)):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	515 continue
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	516
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	517 idstocheck.append((file, bin(id)))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	518
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	519 datastore = self.datastore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	520 historystore = self.historystore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	521 if force:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	522 datastore = contentstore.unioncontentstore(*repo.shareddatastores)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	523 historystore = metadatastore.unionmetadatastore(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	524 *repo.sharedhistorystores)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	525
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	526 missingids = set()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	527 if fetchdata:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	528 missingids.update(datastore.getmissing(idstocheck))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	529 if fetchhistory:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	530 missingids.update(historystore.getmissing(idstocheck))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	531
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	532 # partition missing nodes into nullid and not-nullid so we can
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	533 # warn about this filtering potentially shadowing bugs.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	534 nullids = len([None for unused, id in missingids if id == nullid])
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	535 if nullids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	536 missingids = [(f, id) for f, id in missingids if id != nullid]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	537 repo.ui.develwarn(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	538 ('remotefilelog not fetching %d null revs'
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	539 ' - this is likely hiding bugs' % nullids),
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	540 config='remotefilelog-ext')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	541 if missingids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	542 global fetches, fetched, fetchcost
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	543 fetches += 1
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	544
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	545 # We want to be able to detect excess individual file downloads, so
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	546 # let's log that information for debugging.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	547 if fetches >= 15 and fetches < 18:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	548 if fetches == 15:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	549 fetchwarning = self.ui.config('remotefilelog',
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	550 'fetchwarning')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	551 if fetchwarning:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	552 self.ui.warn(fetchwarning + '\n')
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	553 self.logstacktrace()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	554 missingids = [(file, hex(id)) for file, id in missingids]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	555 fetched += len(missingids)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	556 start = time.time()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	557 missingids = self.request(missingids)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	558 if missingids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	559 raise error.Abort(_("unable to download %d files") %
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	560 len(missingids))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	561 fetchcost += time.time() - start
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	562 self._lfsprefetch(fileids)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	563
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	564 def _lfsprefetch(self, fileids):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	565 if not _lfsmod or not util.safehasattr(
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	566 self.repo.svfs, 'lfslocalblobstore'):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	567 return
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	568 if not _lfsmod.wrapper.candownload(self.repo):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	569 return
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	570 pointers = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	571 store = self.repo.svfs.lfslocalblobstore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	572 for file, id in fileids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	573 node = bin(id)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	574 rlog = self.repo.file(file)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	575 if rlog.flags(node) & revlog.REVIDX_EXTSTORED:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	576 text = rlog.revision(node, raw=True)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	577 p = _lfsmod.pointer.deserialize(text)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	578 oid = p.oid()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	579 if not store.has(oid):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	580 pointers.append(p)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	581 if len(pointers) > 0:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	582 self.repo.svfs.lfsremoteblobstore.readbatch(pointers, store)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	583 assert all(store.has(p.oid()) for p in pointers)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	584
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	585 def logstacktrace(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	586 import traceback
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	587 self.ui.log('remotefilelog', 'excess remotefilelog fetching:\n%s\n',
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental Augie Fackler <augie@google.com> parents: diff changeset	588 ''.join(traceback.format_stack()))

Mercurial > public > mercurial-scm > hg-stable

annotate hgext/remotefilelog/fileserverclient.py @ 40865:b34b1b865ef9