Mercurial > public > mercurial-scm > hg-stable

annotate hgext/remotefilelog/fileserverclient.py @ 46114:59fa3890d40a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
node: import symbols explicitly There is no point in lazy importing mercurial.node, it is used all over the place anyway. So consistently import the used symbols directly. Fix one file using symbols indirectly via mercurial.revlog. Differential Revision: https://phab.mercurial-scm.org/D9480
author Joerg Sonnenberger <joerg@bec.de>
date Tue, 01 Dec 2020 21:54:46 +0100
parents 89a2afe31e82
children d55b71393907
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
1 # fileserverclient.py - client for communicating with the cache process
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
2 #
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
3 # Copyright 2013 Facebook, Inc.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
4 #
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
7
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
8 from __future__ import absolute_import
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
9
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
10 import io
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
11 import os
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
12 import threading
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
13 import time
40557
e2a1584e9e3f remotefilelog: rip out lz4 support
Augie Fackler <augie@google.com>
parents: 40552
diff changeset
14 import zlib
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
15
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
16 from mercurial.i18n import _
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
17 from mercurial.node import bin, hex, nullid
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
18 from mercurial import (
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
19 error,
40657
13d4ad8d7801 py3: fix keyword arguments handling in hgext/remotefilelog/
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40561
diff changeset
20 pycompat,
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
21 revlog,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
22 sshpeer,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
23 util,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
24 wireprotov1peer,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
25 )
44062
2d49482d0dd4 hgext: replace references to hashlib.sha1 with hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44004
diff changeset
26 from mercurial.utils import (
2d49482d0dd4 hgext: replace references to hashlib.sha1 with hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44004
diff changeset
27 hashutil,
2d49482d0dd4 hgext: replace references to hashlib.sha1 with hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44004
diff changeset
28 procutil,
2d49482d0dd4 hgext: replace references to hashlib.sha1 with hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44004
diff changeset
29 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
30
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
31 from . import (
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
32 constants,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
33 contentstore,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
34 metadatastore,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
35 )
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
36
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
37 _sshv1peer = sshpeer.sshv1peer
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
38
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
39 # Statistics for debugging
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
40 fetchcost = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
41 fetches = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
42 fetched = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
43 fetchmisses = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
44
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
45 _lfsmod = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
46
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
47
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
48 def getcachekey(reponame, file, id):
46114
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45957
diff changeset
49 pathhash = hex(hashutil.sha1(file).digest())
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
50 return os.path.join(reponame, pathhash[:2], pathhash[2:], id)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
51
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
52
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
53 def getlocalkey(file, id):
46114
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45957
diff changeset
54 pathhash = hex(hashutil.sha1(file).digest())
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
55 return os.path.join(pathhash, id)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
56
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
57
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
58 def peersetup(ui, peer):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
59 class remotefilepeer(peer.__class__):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
60 @wireprotov1peer.batchable
40561
466dd4d70bff remotefilelog: rename wireproto methods and remaining capabilities
Augie Fackler <augie@google.com>
parents: 40560
diff changeset
61 def x_rfl_getfile(self, file, node):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
62 if not self.capable(b'x_rfl_getfile'):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
63 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
64 b'configured remotefile server does not support getfile'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
65 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
66 f = wireprotov1peer.future()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
67 yield {b'file': file, b'node': node}, f
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
68 code, data = f.value.split(b'\0', 1)
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
69 if int(code):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
70 raise error.LookupError(file, node, data)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
71 yield data
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
72
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
73 @wireprotov1peer.batchable
40561
466dd4d70bff remotefilelog: rename wireproto methods and remaining capabilities
Augie Fackler <augie@google.com>
parents: 40560
diff changeset
74 def x_rfl_getflogheads(self, path):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
75 if not self.capable(b'x_rfl_getflogheads'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
76 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
77 b'configured remotefile server does not '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
78 b'support getflogheads'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
79 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
80 f = wireprotov1peer.future()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
81 yield {b'path': path}, f
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
82 heads = f.value.split(b'\n') if f.value else []
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
83 yield heads
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
84
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
85 def _updatecallstreamopts(self, command, opts):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
86 if command != b'getbundle':
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
87 return
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
88 if (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
89 constants.NETWORK_CAP_LEGACY_SSH_GETFILES
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
90 not in self.capabilities()
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
91 ):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
92 return
43115
4aa72cdf616f py3: delete b'' prefix from safehasattr arguments
Martin von Zweigbergk <martinvonz@google.com>
parents: 43077
diff changeset
93 if not util.safehasattr(self, '_localrepo'):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
94 return
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
95 if (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
96 constants.SHALLOWREPO_REQUIREMENT
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
97 not in self._localrepo.requirements
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
98 ):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
99 return
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
100
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
101 bundlecaps = opts.get(b'bundlecaps')
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
102 if bundlecaps:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
103 bundlecaps = [bundlecaps]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
104 else:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
105 bundlecaps = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
106
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
107 # shallow, includepattern, and excludepattern are a hacky way of
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
108 # carrying over data from the local repo to this getbundle
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
109 # command. We need to do it this way because bundle1 getbundle
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
110 # doesn't provide any other place we can hook in to manipulate
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
111 # getbundle args before it goes across the wire. Once we get rid
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
112 # of bundle1, we can use bundle2's _pullbundle2extraprepare to
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
113 # do this more cleanly.
40559
ed19958dbf5d remotefilelog: consolidate and rename bundle2 capability
Augie Fackler <augie@google.com>
parents: 40558
diff changeset
114 bundlecaps.append(constants.BUNDLE2_CAPABLITY)
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
115 if self._localrepo.includepattern:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
116 patterns = b'\0'.join(self._localrepo.includepattern)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
117 includecap = b"includepattern=" + patterns
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
118 bundlecaps.append(includecap)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
119 if self._localrepo.excludepattern:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
120 patterns = b'\0'.join(self._localrepo.excludepattern)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
121 excludecap = b"excludepattern=" + patterns
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
122 bundlecaps.append(excludecap)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
123 opts[b'bundlecaps'] = b','.join(bundlecaps)
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
124
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
125 def _sendrequest(self, command, args, **opts):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
126 self._updatecallstreamopts(command, args)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
127 return super(remotefilepeer, self)._sendrequest(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
128 command, args, **opts
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
129 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
130
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
131 def _callstream(self, command, **opts):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
132 supertype = super(remotefilepeer, self)
43115
4aa72cdf616f py3: delete b'' prefix from safehasattr arguments
Martin von Zweigbergk <martinvonz@google.com>
parents: 43077
diff changeset
133 if not util.safehasattr(supertype, '_sendrequest'):
40657
13d4ad8d7801 py3: fix keyword arguments handling in hgext/remotefilelog/
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40561
diff changeset
134 self._updatecallstreamopts(command, pycompat.byteskwargs(opts))
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
135 return super(remotefilepeer, self)._callstream(command, **opts)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
136
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
137 peer.__class__ = remotefilepeer
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
138
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
139
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
140 class cacheconnection(object):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
141 """The connection for communicating with the remote cache. Performs
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
142 gets and sets by communicating with an external process that has the
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
143 cache-specific implementation.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
144 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
145
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
146 def __init__(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
147 self.pipeo = self.pipei = self.pipee = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
148 self.subprocess = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
149 self.connected = False
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
150
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
151 def connect(self, cachecommand):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
152 if self.pipeo:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
153 raise error.Abort(_(b"cache connection already open"))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
154 self.pipei, self.pipeo, self.pipee, self.subprocess = procutil.popen4(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
155 cachecommand
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
156 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
157 self.connected = True
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
158
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
159 def close(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
160 def tryclose(pipe):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
161 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
162 pipe.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
163 except Exception:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
164 pass
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
165
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
166 if self.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
167 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
168 self.pipei.write(b"exit\n")
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
169 except Exception:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
170 pass
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
171 tryclose(self.pipei)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
172 self.pipei = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
173 tryclose(self.pipeo)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
174 self.pipeo = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
175 tryclose(self.pipee)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
176 self.pipee = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
177 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
178 # Wait for process to terminate, making sure to avoid deadlock.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
179 # See https://docs.python.org/2/library/subprocess.html for
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
180 # warnings about wait() and deadlocking.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
181 self.subprocess.communicate()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
182 except Exception:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
183 pass
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
184 self.subprocess = None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
185 self.connected = False
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
186
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
187 def request(self, request, flush=True):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
188 if self.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
189 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
190 self.pipei.write(request)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
191 if flush:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
192 self.pipei.flush()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
193 except IOError:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
194 self.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
195
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
196 def receiveline(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
197 if not self.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
198 return None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
199 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
200 result = self.pipeo.readline()[:-1]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
201 if not result:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
202 self.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
203 except IOError:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
204 self.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
205
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
206 return result
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
207
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
208
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
209 def _getfilesbatch(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
210 remote, receivemissing, progresstick, missed, idmap, batchsize
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
211 ):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
212 # Over http(s), iterbatch is a streamy method and we can start
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
213 # looking at results early. This means we send one (potentially
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
214 # large) request, but then we show nice progress as we process
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
215 # file results, rather than showing chunks of $batchsize in
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
216 # progress.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
217 #
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
218 # Over ssh, iterbatch isn't streamy because batch() wasn't
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
219 # explicitly designed as a streaming method. In the future we
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
220 # should probably introduce a streambatch() method upstream and
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
221 # use that for this.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
222 with remote.commandexecutor() as e:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
223 futures = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
224 for m in missed:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
225 futures.append(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
226 e.callcommand(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
227 b'x_rfl_getfile', {b'file': idmap[m], b'node': m[-40:]}
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
228 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
229 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
230
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
231 for i, m in enumerate(missed):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
232 r = futures[i].result()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
233 futures[i] = None # release memory
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
234 file_ = idmap[m]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
235 node = m[-40:]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
236 receivemissing(io.BytesIO(b'%d\n%s' % (len(r), r)), file_, node)
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
237 progresstick()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
238
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
239
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
240 def _getfiles_optimistic(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
241 remote, receivemissing, progresstick, missed, idmap, step
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
242 ):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
243 remote._callstream(b"x_rfl_getfiles")
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
244 i = 0
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
245 pipeo = remote._pipeo
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
246 pipei = remote._pipei
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
247 while i < len(missed):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
248 # issue a batch of requests
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
249 start = i
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
250 end = min(len(missed), start + step)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
251 i = end
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
252 for missingid in missed[start:end]:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
253 # issue new request
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
254 versionid = missingid[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
255 file = idmap[missingid]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
256 sshrequest = b"%s%s\n" % (versionid, file)
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
257 pipeo.write(sshrequest)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
258 pipeo.flush()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
259
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
260 # receive batch results
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
261 for missingid in missed[start:end]:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
262 versionid = missingid[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
263 file = idmap[missingid]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
264 receivemissing(pipei, file, versionid)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
265 progresstick()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
266
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
267 # End the command
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
268 pipeo.write(b'\n')
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
269 pipeo.flush()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
270
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
271
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
272 def _getfiles_threaded(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
273 remote, receivemissing, progresstick, missed, idmap, step
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
274 ):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
275 remote._callstream(b"getfiles")
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
276 pipeo = remote._pipeo
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
277 pipei = remote._pipei
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
278
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
279 def writer():
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
280 for missingid in missed:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
281 versionid = missingid[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
282 file = idmap[missingid]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
283 sshrequest = b"%s%s\n" % (versionid, file)
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
284 pipeo.write(sshrequest)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
285 pipeo.flush()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
286
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
287 writerthread = threading.Thread(target=writer)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
288 writerthread.daemon = True
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
289 writerthread.start()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
290
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
291 for missingid in missed:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
292 versionid = missingid[-40:]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
293 file = idmap[missingid]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
294 receivemissing(pipei, file, versionid)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
295 progresstick()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
296
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
297 writerthread.join()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
298 # End the command
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
299 pipeo.write(b'\n')
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
300 pipeo.flush()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
301
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
302
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
303 class fileserverclient(object):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44062
diff changeset
304 """A client for requesting files from the remote file server."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
305
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
306 def __init__(self, repo):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
307 ui = repo.ui
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
308 self.repo = repo
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
309 self.ui = ui
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
310 self.cacheprocess = ui.config(b"remotefilelog", b"cacheprocess")
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
311 if self.cacheprocess:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
312 self.cacheprocess = util.expandpath(self.cacheprocess)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
313
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
314 # This option causes remotefilelog to pass the full file path to the
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
315 # cacheprocess instead of a hashed key.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
316 self.cacheprocesspasspath = ui.configbool(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
317 b"remotefilelog", b"cacheprocess.includepath"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
318 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
319
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
320 self.debugoutput = ui.configbool(b"remotefilelog", b"debug")
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
321
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
322 self.remotecache = cacheconnection()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
323
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
324 def setstore(self, datastore, historystore, writedata, writehistory):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
325 self.datastore = datastore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
326 self.historystore = historystore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
327 self.writedata = writedata
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
328 self.writehistory = writehistory
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
329
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
330 def _connect(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
331 return self.repo.connectionpool.get(self.repo.fallbackpath)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
332
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
333 def request(self, fileids):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
334 """Takes a list of filename/node pairs and fetches them from the
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
335 server. Files are stored in the local cache.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
336 A list of nodes that the server couldn't find is returned.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
337 If the connection fails, an exception is raised.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
338 """
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
339 if not self.remotecache.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
340 self.connect()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
341 cache = self.remotecache
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
342 writedata = self.writedata
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
343
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
344 repo = self.repo
40865
b34b1b865ef9 remotefilelog: avoid temporarily using "count" variable as synonym for "total"
Martin von Zweigbergk <martinvonz@google.com>
parents: 40864
diff changeset
345 total = len(fileids)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
346 request = b"get\n%d\n" % total
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
347 idmap = {}
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
348 reponame = repo.name
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
349 for file, id in fileids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
350 fullid = getcachekey(reponame, file, id)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
351 if self.cacheprocesspasspath:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
352 request += file + b'\0'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
353 request += fullid + b"\n"
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
354 idmap[fullid] = file
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
355
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
356 cache.request(request)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
357
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
358 progress = self.ui.makeprogress(_(b'downloading'), total=total)
40864
e58cd7ede1c3 remotefilelog: use progress helper in fileserverclient
Martin von Zweigbergk <martinvonz@google.com>
parents: 40661
diff changeset
359 progress.update(0)
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
360
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
361 missed = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
362 while True:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
363 missingid = cache.receiveline()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
364 if not missingid:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
365 missedset = set(missed)
40660
9769e0f6ffe0 py3: don't use dict.iterkeys()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40659
diff changeset
366 for missingid in idmap:
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
367 if not missingid in missedset:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
368 missed.append(missingid)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
369 self.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
370 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
371 b"warning: cache connection closed early - "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
372 + b"falling back to server\n"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
373 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
374 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
375 break
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
376 if missingid == b"0":
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
377 break
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
378 if missingid.startswith(b"_hits_"):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
379 # receive progress reports
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
380 parts = missingid.split(b"_")
40869
74e3df766052 remotefilelog: rely on progress helper for keeping track of position
Martin von Zweigbergk <martinvonz@google.com>
parents: 40868
diff changeset
381 progress.increment(int(parts[2]))
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
382 continue
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
383
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
384 missed.append(missingid)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
385
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
386 global fetchmisses
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
387 fetchmisses += len(missed)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
388
40867
fe0b65ff00e7 remotefilelog: replace a "a=[expr]; b=a[0]" by "b=expr; a = [b]"
Martin von Zweigbergk <martinvonz@google.com>
parents: 40866
diff changeset
389 fromcache = total - len(missed)
40868
fcee112fb7c4 remotefilelog: reduce use of "count" container
Martin von Zweigbergk <martinvonz@google.com>
parents: 40867
diff changeset
390 progress.update(fromcache, total=total)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
391 self.ui.log(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
392 b"remotefilelog",
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
393 b"remote cache hit rate is %r of %r\n",
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
394 fromcache,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
395 total,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
396 hit=fromcache,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
397 total=total,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
398 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
399
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
400 oldumask = os.umask(0o002)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
401 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
402 # receive cache misses from master
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
403 if missed:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
404 # When verbose is true, sshpeer prints 'running ssh...'
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
405 # to stdout, which can interfere with some command
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
406 # outputs
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
407 verbose = self.ui.verbose
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
408 self.ui.verbose = False
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
409 try:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
410 with self._connect() as conn:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
411 remote = conn.peer
40558
354acd0dc637 remotefilelog: rename capability for legacy ssh file fetching method
Augie Fackler <augie@google.com>
parents: 40557
diff changeset
412 if remote.capable(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
413 constants.NETWORK_CAP_LEGACY_SSH_GETFILES
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
414 ):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
415 if not isinstance(remote, _sshv1peer):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
416 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43115
diff changeset
417 b'remotefilelog requires ssh servers'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
418 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
419 step = self.ui.configint(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
420 b'remotefilelog', b'getfilesstep'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
421 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
422 getfilestype = self.ui.config(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
423 b'remotefilelog', b'getfilestype'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
424 )
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
425 if getfilestype == b'threaded':
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
426 _getfiles = _getfiles_threaded
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
427 else:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
428 _getfiles = _getfiles_optimistic
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
429 _getfiles(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
430 remote,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
431 self.receivemissing,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
432 progress.increment,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
433 missed,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
434 idmap,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
435 step,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
436 )
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
437 elif remote.capable(b"x_rfl_getfile"):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
438 if remote.capable(b'batch'):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
439 batchdefault = 100
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
440 else:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
441 batchdefault = 10
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
442 batchsize = self.ui.configint(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
443 b'remotefilelog', b'batchsize', batchdefault
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
444 )
42292
65f3a77223e0 remotefilelog: log when we're about to fetch files
Augie Fackler <augie@google.com>
parents: 42036
diff changeset
445 self.ui.debug(
65f3a77223e0 remotefilelog: log when we're about to fetch files
Augie Fackler <augie@google.com>
parents: 42036
diff changeset
446 b'requesting %d files from '
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
447 b'remotefilelog server...\n' % len(missed)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
448 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
449 _getfilesbatch(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
450 remote,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
451 self.receivemissing,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
452 progress.increment,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
453 missed,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
454 idmap,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
455 batchsize,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
456 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
457 else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
458 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
459 b"configured remotefilelog server"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
460 b" does not support remotefilelog"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
461 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
462
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
463 self.ui.log(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
464 b"remotefilefetchlog",
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
465 b"Success\n",
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
466 fetched_files=progress.pos - fromcache,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
467 total_to_fetch=total - fromcache,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
468 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
469 except Exception:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
470 self.ui.log(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
471 b"remotefilefetchlog",
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
472 b"Fail\n",
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
473 fetched_files=progress.pos - fromcache,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
474 total_to_fetch=total - fromcache,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
475 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
476 raise
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
477 finally:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
478 self.ui.verbose = verbose
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
479 # send to memcache
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
480 request = b"set\n%d\n%s\n" % (len(missed), b"\n".join(missed))
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
481 cache.request(request)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
482
40864
e58cd7ede1c3 remotefilelog: use progress helper in fileserverclient
Martin von Zweigbergk <martinvonz@google.com>
parents: 40661
diff changeset
483 progress.complete()
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
484
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
485 # mark ourselves as a user of this cache
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
486 writedata.markrepo(self.repo.path)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
487 finally:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
488 os.umask(oldumask)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
489
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
490 def receivemissing(self, pipe, filename, node):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
491 line = pipe.readline()[:-1]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
492 if not line:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
493 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
494 _(b"error downloading file contents:"),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
495 _(b"connection closed early"),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
496 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
497 size = int(line)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
498 data = pipe.read(size)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
499 if len(data) != size:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
500 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
501 _(b"error downloading file contents:"),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
502 _(b"only received %s of %s bytes") % (len(data), size),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
503 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
504
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
505 self.writedata.addremotefilelognode(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
506 filename, bin(node), zlib.decompress(data)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
507 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
508
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
509 def connect(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
510 if self.cacheprocess:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
511 cmd = b"%s %s" % (self.cacheprocess, self.writedata._path)
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
512 self.remotecache.connect(cmd)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
513 else:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
514 # If no cache process is specified, we fake one that always
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
515 # returns cache misses. This enables tests to run easily
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
516 # and may eventually allow us to be a drop in replacement
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
517 # for the largefiles extension.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
518 class simplecache(object):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
519 def __init__(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
520 self.missingids = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
521 self.connected = True
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
522
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
523 def close(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
524 pass
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
525
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
526 def request(self, value, flush=True):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
527 lines = value.split(b"\n")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
528 if lines[0] != b"get":
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
529 return
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
530 self.missingids = lines[2:-1]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
531 self.missingids.append(b'0')
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
532
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
533 def receiveline(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
534 if len(self.missingids) > 0:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
535 return self.missingids.pop(0)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
536 return None
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
537
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
538 self.remotecache = simplecache()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
539
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
540 def close(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
541 if fetches:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
542 msg = (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
543 b"%d files fetched over %d fetches - "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
544 + b"(%d misses, %0.2f%% hit ratio) over %0.2fs\n"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
545 ) % (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
546 fetched,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
547 fetches,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
548 fetchmisses,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
549 float(fetched - fetchmisses) / float(fetched) * 100.0,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
550 fetchcost,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
551 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
552 if self.debugoutput:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
553 self.ui.warn(msg)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
554 self.ui.log(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
555 b"remotefilelog.prefetch",
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
556 msg.replace(b"%", b"%%"),
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
557 remotefilelogfetched=fetched,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
558 remotefilelogfetches=fetches,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
559 remotefilelogfetchmisses=fetchmisses,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
560 remotefilelogfetchtime=fetchcost * 1000,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
561 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
562
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
563 if self.remotecache.connected:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
564 self.remotecache.close()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
565
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
566 def prefetch(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
567 self, fileids, force=False, fetchdata=True, fetchhistory=False
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
568 ):
45957
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44062
diff changeset
569 """downloads the given file versions to the cache"""
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
570 repo = self.repo
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
571 idstocheck = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
572 for file, id in fileids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
573 # hack
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
574 # - we don't use .hgtags
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
575 # - workingctx produces ids with length 42,
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
576 # which we skip since they aren't in any cache
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
577 if (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
578 file == b'.hgtags'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
579 or len(id) == 42
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
580 or not repo.shallowmatch(file)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
581 ):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
582 continue
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
583
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
584 idstocheck.append((file, bin(id)))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
585
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
586 datastore = self.datastore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
587 historystore = self.historystore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
588 if force:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
589 datastore = contentstore.unioncontentstore(*repo.shareddatastores)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
590 historystore = metadatastore.unionmetadatastore(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
591 *repo.sharedhistorystores
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
592 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
593
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
594 missingids = set()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
595 if fetchdata:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
596 missingids.update(datastore.getmissing(idstocheck))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
597 if fetchhistory:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
598 missingids.update(historystore.getmissing(idstocheck))
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
599
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
600 # partition missing nodes into nullid and not-nullid so we can
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
601 # warn about this filtering potentially shadowing bugs.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
602 nullids = len([None for unused, id in missingids if id == nullid])
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
603 if nullids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
604 missingids = [(f, id) for f, id in missingids if id != nullid]
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
605 repo.ui.develwarn(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
606 (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
607 b'remotefilelog not fetching %d null revs'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
608 b' - this is likely hiding bugs' % nullids
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
609 ),
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
610 config=b'remotefilelog-ext',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
611 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
612 if missingids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
613 global fetches, fetched, fetchcost
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
614 fetches += 1
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
615
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
616 # We want to be able to detect excess individual file downloads, so
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
617 # let's log that information for debugging.
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
618 if fetches >= 15 and fetches < 18:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
619 if fetches == 15:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
620 fetchwarning = self.ui.config(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
621 b'remotefilelog', b'fetchwarning'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
622 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
623 if fetchwarning:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
624 self.ui.warn(fetchwarning + b'\n')
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
625 self.logstacktrace()
42036
0129bf02fa04 remotefilelog: prefetch files in deterministic order
Martin von Zweigbergk <martinvonz@google.com>
parents: 41768
diff changeset
626 missingids = [(file, hex(id)) for file, id in sorted(missingids)]
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
627 fetched += len(missingids)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
628 start = time.time()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
629 missingids = self.request(missingids)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
630 if missingids:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
631 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
632 _(b"unable to download %d files") % len(missingids)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
633 )
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
634 fetchcost += time.time() - start
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
635 self._lfsprefetch(fileids)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
636
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
637 def _lfsprefetch(self, fileids):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
638 if not _lfsmod or not util.safehasattr(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
639 self.repo.svfs, b'lfslocalblobstore'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
640 ):
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
641 return
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
642 if not _lfsmod.wrapper.candownload(self.repo):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
643 return
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
644 pointers = []
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
645 store = self.repo.svfs.lfslocalblobstore
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
646 for file, id in fileids:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
647 node = bin(id)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
648 rlog = self.repo.file(file)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
649 if rlog.flags(node) & revlog.REVIDX_EXTSTORED:
42787
fdeb4c1d23d5 rawdata: update caller in remotefilelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 42292
diff changeset
650 text = rlog.rawdata(node)
40545
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
651 p = _lfsmod.pointer.deserialize(text)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
652 oid = p.oid()
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
653 if not store.has(oid):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
654 pointers.append(p)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
655 if len(pointers) > 0:
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
656 self.repo.svfs.lfsremoteblobstore.readbatch(pointers, store)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
657 assert all(store.has(p.oid()) for p in pointers)
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
658
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
659 def logstacktrace(self):
3a333a582d7b remotefilelog: import pruned-down remotefilelog extension from hg-experimental
Augie Fackler <augie@google.com>
parents:
diff changeset
660 import traceback
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
661
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
662 self.ui.log(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
663 b'remotefilelog',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
664 b'excess remotefilelog fetching:\n%s\n',
44004
faf00e30b63f remotefilelog: actually fix (and test this time) a bytes vs str issue
Kyle Lippincott <spectral@google.com>
parents: 43853
diff changeset
665 b''.join(pycompat.sysbytes(s) for s in traceback.format_stack()),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42787
diff changeset
666 )