Mercurial > public > mercurial-scm > hg-stable
annotate mercurial/exchangev2.py @ 39654:d292328e0143
exchangev2: fetch manifest revisions
Now that the server has support for retrieving manifest data, we can
implement the client bits to call it.
We teach the changeset fetching code to capture the manifest revisions
that are encountered on incoming changesets. We then feed this into a
new function which filters out known manifests and then batches up
manifest data requests to the server.
This is different from the previous wire protocol in a few notable
ways.
First, the client fetches manifest data separately and explicitly.
Before, we'd ask the server for data pertaining to some changesets
(via a "getbundle" command) and manifests (and files) would be sent
automatically. Providing an API for looking up just manifest data
separately gives clients much more flexibility for manifest management.
For example, a client may choose to only fetch manifest data on demand
instead of prefetching it (i.e. partial clone).
Second, we send N commands to the server for manifest retrieval instead
of 1. This property has a few nice side-effects. One is that the
deterministic nature of the requests lends itself to server-side
caching. For example, say the remote has 50,000 manifests. If the
server is configured to cache responses, each time a new commit
arrives, you will have a cache miss and need to regenerate all outgoing
data. But if you makes N requests requesting 10,000 manifests each,
a new commit will still yield cache hits on the initial, unchanged
manifest batches/requests.
A derived benefit from these properties is that resumable clone is
conceptually simpler to implement. When making a monolithic request
for all of the repository data, recovering from an interrupted clone
is hard because the server was in the driver's seat and was maintaining
state about all the data that needed transferred. With the client
driving fetching, the client can persist the set of unfetched entities
and retry/resume a fetch if something goes wrong. Or we can fetch all
data N changesets at a time and slowly build up a repository. This
approach is drastically easier to implement when we have server APIs
exposing low-level repository primitives (such as manifests and files).
We don't yet support tree manifests. But it should be possible to
implement that with the existing wire protocol command.
Differential Revision: https://phab.mercurial-scm.org/D4489
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Wed, 05 Sep 2018 09:09:57 -0700 |
parents | 399ddd3227a4 |
children | 039bf1eddc2e |
rev | line source |
---|---|
39645
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
1 # exchangev2.py - repository exchange for wire protocol version 2 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
2 # |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com> |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
4 # |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
7 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
8 from __future__ import absolute_import |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
9 |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
10 import weakref |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
11 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
12 from .i18n import _ |
39645
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
13 from .node import ( |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
14 nullid, |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
15 short, |
39645
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
16 ) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
17 from . import ( |
39651
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
18 bookmarks, |
39654
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
19 error, |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
20 mdiff, |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
21 phases, |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
22 pycompat, |
39645
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
23 setdiscovery, |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
24 ) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
25 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
26 def pull(pullop): |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
27 """Pull using wire protocol version 2.""" |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
28 repo = pullop.repo |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
29 remote = pullop.remote |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
30 tr = pullop.trmanager.transaction() |
39645
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
31 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
32 # Figure out what needs to be fetched. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
33 common, fetch, remoteheads = _pullchangesetdiscovery( |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
34 repo, remote, pullop.heads, abortwhenunrelated=pullop.force) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
35 |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
36 # And fetch the data. |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
37 pullheads = pullop.heads or remoteheads |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
38 csetres = _fetchchangesets(repo, tr, remote, common, fetch, pullheads) |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
39 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
40 # New revisions are written to the changelog. But all other updates |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
41 # are deferred. Do those now. |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
42 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
43 # Ensure all new changesets are draft by default. If the repo is |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
44 # publishing, the phase will be adjusted by the loop below. |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
45 if csetres['added']: |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
46 phases.registernew(repo, tr, phases.draft, csetres['added']) |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
47 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
48 # And adjust the phase of all changesets accordingly. |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
49 for phase in phases.phasenames: |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
50 if phase == b'secret' or not csetres['nodesbyphase'][phase]: |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
51 continue |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
52 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
53 phases.advanceboundary(repo, tr, phases.phasenames.index(phase), |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
54 csetres['nodesbyphase'][phase]) |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
55 |
39651
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
56 # Write bookmark updates. |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
57 bookmarks.updatefromremote(repo.ui, repo, csetres['bookmarks'], |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
58 remote.url(), pullop.gettransaction, |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
59 explicit=pullop.explicitbookmarks) |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
60 |
39654
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
61 _fetchmanifests(repo, tr, remote, csetres['manifestnodes']) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
62 |
39645
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
63 def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True): |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
64 """Determine which changesets need to be pulled.""" |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
65 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
66 if heads: |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
67 knownnode = repo.changelog.hasnode |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
68 if all(knownnode(head) for head in heads): |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
69 return heads, False, heads |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
70 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
71 # TODO wire protocol version 2 is capable of more efficient discovery |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
72 # than setdiscovery. Consider implementing something better. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
73 common, fetch, remoteheads = setdiscovery.findcommonheads( |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
74 repo.ui, repo, remote, abortwhenunrelated=abortwhenunrelated) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
75 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
76 common = set(common) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
77 remoteheads = set(remoteheads) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
78 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
79 # If a remote head is filtered locally, put it back in the common set. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
80 # See the comment in exchange._pulldiscoverychangegroup() for more. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
81 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
82 if fetch and remoteheads: |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
83 nodemap = repo.unfiltered().changelog.nodemap |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
84 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
85 common |= {head for head in remoteheads if head in nodemap} |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
86 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
87 if set(remoteheads).issubset(common): |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
88 fetch = [] |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
89 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
90 common.discard(nullid) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
91 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
92 return common, fetch, remoteheads |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
93 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
94 def _fetchchangesets(repo, tr, remote, common, fetch, remoteheads): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
95 # TODO consider adding a step here where we obtain the DAG shape first |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
96 # (or ask the server to slice changesets into chunks for us) so that |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
97 # we can perform multiple fetches in batches. This will facilitate |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
98 # resuming interrupted clones, higher server-side cache hit rates due |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
99 # to smaller segments, etc. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
100 with remote.commandexecutor() as e: |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
101 objs = e.callcommand(b'changesetdata', { |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
102 b'noderange': [sorted(common), sorted(remoteheads)], |
39651
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
103 b'fields': {b'bookmarks', b'parents', b'phase', b'revision'}, |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
104 }).result() |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
105 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
106 # The context manager waits on all response data when exiting. So |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
107 # we need to remain in the context manager in order to stream data. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
108 return _processchangesetdata(repo, tr, objs) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
109 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
110 def _processchangesetdata(repo, tr, objs): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
111 repo.hook('prechangegroup', throw=True, |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
112 **pycompat.strkwargs(tr.hookargs)) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
113 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
114 urepo = repo.unfiltered() |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
115 cl = urepo.changelog |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
116 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
117 cl.delayupdate(tr) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
118 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
119 # The first emitted object is a header describing the data that |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
120 # follows. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
121 meta = next(objs) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
122 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
123 progress = repo.ui.makeprogress(_('changesets'), |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
124 unit=_('chunks'), |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
125 total=meta.get(b'totalitems')) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
126 |
39654
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
127 manifestnodes = {} |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
128 |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
129 def linkrev(node): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
130 repo.ui.debug('add changeset %s\n' % short(node)) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
131 # Linkrev for changelog is always self. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
132 return len(cl) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
133 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
134 def onchangeset(cl, node): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
135 progress.increment() |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
136 |
39654
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
137 revision = cl.changelogrevision(node) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
138 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
139 # We need to preserve the mapping of changelog revision to node |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
140 # so we can set the linkrev accordingly when manifests are added. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
141 manifestnodes[cl.rev(node)] = revision.manifest |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
142 |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
143 nodesbyphase = {phase: set() for phase in phases.phasenames} |
39651
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
144 remotebookmarks = {} |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
145 |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
146 # addgroup() expects a 7-tuple describing revisions. This normalizes |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
147 # the wire data to that format. |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
148 # |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
149 # This loop also aggregates non-revision metadata, such as phase |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
150 # data. |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
151 def iterrevisions(): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
152 for cset in objs: |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
153 node = cset[b'node'] |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
154 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
155 if b'phase' in cset: |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
156 nodesbyphase[cset[b'phase']].add(node) |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
157 |
39651
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
158 for mark in cset.get(b'bookmarks', []): |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
159 remotebookmarks[mark] = node |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
160 |
39652
399ddd3227a4
wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39651
diff
changeset
|
161 # TODO add mechanism for extensions to examine records so they |
399ddd3227a4
wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39651
diff
changeset
|
162 # can siphon off custom data fields. |
399ddd3227a4
wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39651
diff
changeset
|
163 |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
164 # Some entries might only be metadata only updates. |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
165 if b'revisionsize' not in cset: |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
166 continue |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
167 |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
168 data = next(objs) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
169 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
170 yield ( |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
171 node, |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
172 cset[b'parents'][0], |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
173 cset[b'parents'][1], |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
174 # Linknode is always itself for changesets. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
175 cset[b'node'], |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
176 # We always send full revisions. So delta base is not set. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
177 nullid, |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
178 mdiff.trivialdiffheader(len(data)) + data, |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
179 # Flags not yet supported. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
180 0, |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
181 ) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
182 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
183 added = cl.addgroup(iterrevisions(), linkrev, weakref.proxy(tr), |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
184 addrevisioncb=onchangeset) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
185 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
186 progress.complete() |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
187 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
188 return { |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
189 'added': added, |
39649
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39647
diff
changeset
|
190 'nodesbyphase': nodesbyphase, |
39651
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39649
diff
changeset
|
191 'bookmarks': remotebookmarks, |
39654
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
192 'manifestnodes': manifestnodes, |
39647
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39645
diff
changeset
|
193 } |
39654
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
194 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
195 def _fetchmanifests(repo, tr, remote, manifestnodes): |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
196 rootmanifest = repo.manifestlog.getstorage(b'') |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
197 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
198 # Some manifests can be shared between changesets. Filter out revisions |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
199 # we already know about. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
200 fetchnodes = [] |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
201 linkrevs = {} |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
202 seen = set() |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
203 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
204 for clrev, node in sorted(manifestnodes.iteritems()): |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
205 if node in seen: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
206 continue |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
207 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
208 try: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
209 rootmanifest.rev(node) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
210 except error.LookupError: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
211 fetchnodes.append(node) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
212 linkrevs[node] = clrev |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
213 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
214 seen.add(node) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
215 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
216 # TODO handle tree manifests |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
217 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
218 # addgroup() expects 7-tuple describing revisions. This normalizes |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
219 # the wire data to that format. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
220 def iterrevisions(objs, progress): |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
221 for manifest in objs: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
222 node = manifest[b'node'] |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
223 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
224 if b'deltasize' in manifest: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
225 basenode = manifest[b'deltabasenode'] |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
226 delta = next(objs) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
227 elif b'revisionsize' in manifest: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
228 basenode = nullid |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
229 revision = next(objs) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
230 delta = mdiff.trivialdiffheader(len(revision)) + revision |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
231 else: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
232 continue |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
233 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
234 yield ( |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
235 node, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
236 manifest[b'parents'][0], |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
237 manifest[b'parents'][1], |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
238 # The value passed in is passed to the lookup function passed |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
239 # to addgroup(). We already have a map of manifest node to |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
240 # changelog revision number. So we just pass in the |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
241 # manifest node here and use linkrevs.__getitem__ as the |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
242 # resolution function. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
243 node, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
244 basenode, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
245 delta, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
246 # Flags not yet supported. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
247 0 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
248 ) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
249 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
250 progress.increment() |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
251 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
252 progress = repo.ui.makeprogress(_('manifests'), unit=_('chunks'), |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
253 total=len(fetchnodes)) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
254 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
255 # Fetch manifests 10,000 per command. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
256 # TODO have server advertise preferences? |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
257 # TODO make size configurable on client? |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
258 batchsize = 10000 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
259 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
260 # We send commands 1 at a time to the remote. This is not the most |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
261 # efficient because we incur a round trip at the end of each batch. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
262 # However, the existing frame-based reactor keeps consuming server |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
263 # data in the background. And this results in response data buffering |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
264 # in memory. This can consume gigabytes of memory. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
265 # TODO send multiple commands in a request once background buffering |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
266 # issues are resolved. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
267 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
268 added = [] |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
269 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
270 for i in pycompat.xrange(0, len(fetchnodes), batchsize): |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
271 batch = [node for node in fetchnodes[i:i + batchsize]] |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
272 if not batch: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
273 continue |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
274 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
275 with remote.commandexecutor() as e: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
276 objs = e.callcommand(b'manifestdata', { |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
277 b'tree': b'', |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
278 b'nodes': batch, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
279 b'fields': {b'parents', b'revision'}, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
280 }).result() |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
281 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
282 # Chomp off header object. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
283 next(objs) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
284 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
285 added.extend(rootmanifest.addgroup( |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
286 iterrevisions(objs, progress), |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
287 linkrevs.__getitem__, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
288 weakref.proxy(tr))) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
289 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
290 progress.complete() |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
291 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
292 return { |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
293 'added': added, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39652
diff
changeset
|
294 } |