mercurial-scm/hg-stable: mercurial/exchangev2.py annotate

annotate mercurial/exchangev2.py @ 46443:0903d6b9b1df

repository: introduce register_changeset callback The new callback is called whenever a changeset is added to the repository (commit, unbundle or exchange). Since the bulk operations already parse the changeset (readfiles or full changesetrevision), always use the latter to avoid redundant lookups. The first consumer of the new interface needs to look at extra. Differential Revision: https://phab.mercurial-scm.org/D9780

author	Joerg Sonnenberger <joerg@bec.de>
date	Tue, 19 Jan 2021 00:20:53 +0100
parents	009130064813
children	711ba0f1057e

rev	line source
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # exchangev2.py - repository exchange for wire protocol version 2
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8 from __future__ import absolute_import
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	10 import collections
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	11 import weakref
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	12
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	13 from .i18n import _
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	14 from .node import (
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	15 nullid,
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	16 short,
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	17 )
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	18 from . import (
39651 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39649 diff changeset	19 bookmarks,
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	20 error,
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	21 mdiff,
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	22 narrowspec,
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	23 phases,
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	24 pycompat,
42823 268662aac075 interfaces: create a new folder for interfaces and move repository.py in it Pulkit Goyal <pulkit@yandex-team.ru> parents: 41217 diff changeset	25 setdiscovery,
268662aac075 interfaces: create a new folder for interfaces and move repository.py in it Pulkit Goyal <pulkit@yandex-team.ru> parents: 41217 diff changeset	26 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	27 from .interfaces import repository
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	28
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30 def pull(pullop):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31 """Pull using wire protocol version 2."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 repo = pullop.repo
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	33 remote = pullop.remote
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	34
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	35 usingrawchangelogandmanifest = _checkuserawstorefiledata(pullop)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	36
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	37 # If this is a clone and it was requested to perform a "stream clone",
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	38 # we obtain the raw files data from the remote then fall back to an
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	39 # incremental pull. This is somewhat hacky and is not nearly robust enough
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	40 # for long-term usage.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	41 if usingrawchangelogandmanifest:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	42 with repo.transaction(b'clone'):
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	43 _fetchrawstorefiles(repo, remote)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	44 repo.invalidate(clearfilecache=True)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	45
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	46 tr = pullop.trmanager.transaction()
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	47
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	48 # We don't use the repo's narrow matcher here because the patterns passed
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	49 # to exchange.pull() could be different.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	50 narrowmatcher = narrowspec.match(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	51 repo.root,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	52 # Empty maps to nevermatcher. So always
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	53 # set includes if missing.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	54 pullop.includepats or {b'path:.'},
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	55 pullop.excludepats,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	56 )
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	57
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	58 if pullop.includepats or pullop.excludepats:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	59 pathfilter = {}
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	60 if pullop.includepats:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	61 pathfilter[b'include'] = sorted(pullop.includepats)
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	62 if pullop.excludepats:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	63 pathfilter[b'exclude'] = sorted(pullop.excludepats)
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	64 else:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	65 pathfilter = None
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	66
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	67 # Figure out what needs to be fetched.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	68 common, fetch, remoteheads = _pullchangesetdiscovery(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	69 repo, remote, pullop.heads, abortwhenunrelated=pullop.force
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	70 )
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	71
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	72 # And fetch the data.
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	73 pullheads = pullop.heads or remoteheads
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	74 csetres = _fetchchangesets(repo, tr, remote, common, fetch, pullheads)
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	75
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	76 # New revisions are written to the changelog. But all other updates
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	77 # are deferred. Do those now.
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	78
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	79 # Ensure all new changesets are draft by default. If the repo is
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	80 # publishing, the phase will be adjusted by the loop below.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	81 if csetres[b'added']:
45813 5d65e04b6a80 phases: convert registernew users to use revision sets Joerg Sonnenberger <joerg@bec.de> parents: 45811 diff changeset	82 phases.registernew(
5d65e04b6a80 phases: convert registernew users to use revision sets Joerg Sonnenberger <joerg@bec.de> parents: 45811 diff changeset	83 repo, tr, phases.draft, [repo[n].rev() for n in csetres[b'added']]
5d65e04b6a80 phases: convert registernew users to use revision sets Joerg Sonnenberger <joerg@bec.de> parents: 45811 diff changeset	84 )
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	85
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	86 # And adjust the phase of all changesets accordingly.
45117 b1e51ef4e536 phases: sparsify phase lists Joerg Sonnenberger <joerg@bec.de> parents: 43594 diff changeset	87 for phasenumber, phase in phases.phasenames.items():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	88 if phase == b'secret' or not csetres[b'nodesbyphase'][phase]:
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	89 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	90
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	91 phases.advanceboundary(
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	92 repo,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	93 tr,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	94 phasenumber,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	95 csetres[b'nodesbyphase'][phase],
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	96 )
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	97
39651 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39649 diff changeset	98 # Write bookmark updates.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	99 bookmarks.updatefromremote(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	100 repo.ui,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	101 repo,
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	102 csetres[b'bookmarks'],
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	103 remote.url(),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	104 pullop.gettransaction,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	105 explicit=pullop.explicitbookmarks,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	106 )
39651 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39649 diff changeset	107
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	108 manres = _fetchmanifests(repo, tr, remote, csetres[b'manifestnodes'])
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	109
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	110 # We don't properly support shallow changeset and manifest yet. So we apply
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	111 # depth limiting locally.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	112 if pullop.depth:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	113 relevantcsetnodes = set()
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	114 clnode = repo.changelog.node
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	115
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	116 for rev in repo.revs(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	117 b'ancestors(%ln, %s)', pullheads, pullop.depth - 1
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	118 ):
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	119 relevantcsetnodes.add(clnode(rev))
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	120
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	121 csetrelevantfilter = lambda n: n in relevantcsetnodes
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	122
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	123 else:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	124 csetrelevantfilter = lambda n: True
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	125
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	126 # If obtaining the raw store files, we need to scan the full repo to
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	127 # derive all the changesets, manifests, and linkrevs.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	128 if usingrawchangelogandmanifest:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	129 csetsforfiles = []
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	130 mnodesforfiles = []
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	131 manifestlinkrevs = {}
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	132
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	133 for rev in repo:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	134 ctx = repo[rev]
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	135 node = ctx.node()
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	136
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	137 if not csetrelevantfilter(node):
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	138 continue
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	139
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	140 mnode = ctx.manifestnode()
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	141
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	142 csetsforfiles.append(node)
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	143 mnodesforfiles.append(mnode)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	144 manifestlinkrevs[mnode] = rev
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	145
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	146 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	147 csetsforfiles = [n for n in csetres[b'added'] if csetrelevantfilter(n)]
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	148 mnodesforfiles = manres[b'added']
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	149 manifestlinkrevs = manres[b'linkrevs']
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	150
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	151 # Find all file nodes referenced by added manifests and fetch those
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	152 # revisions.
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	153 fnodes = _derivefilesfrommanifests(repo, narrowmatcher, mnodesforfiles)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	154 _fetchfilesfromcsets(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	155 repo,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	156 tr,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	157 remote,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	158 pathfilter,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	159 fnodes,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	160 csetsforfiles,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	161 manifestlinkrevs,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	162 shallow=bool(pullop.depth),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	163 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	164
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	165
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	166 def _checkuserawstorefiledata(pullop):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	167 """Check whether we should use rawstorefiledata command to retrieve data."""
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	168
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	169 repo = pullop.repo
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	170 remote = pullop.remote
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	171
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	172 # Command to obtain raw store data isn't available.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	173 if b'rawstorefiledata' not in remote.apidescriptor[b'commands']:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	174 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	175
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	176 # Only honor if user requested stream clone operation.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	177 if not pullop.streamclonerequested:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	178 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	179
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	180 # Only works on empty repos.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	181 if len(repo):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	182 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	183
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	184 # TODO This is super hacky. There needs to be a storage API for this. We
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	185 # also need to check for compatibility with the remote.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	186 if b'revlogv1' not in repo.requirements:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	187 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	188
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	189 return True
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	190
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	191
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	192 def _fetchrawstorefiles(repo, remote):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	193 with remote.commandexecutor() as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	194 objs = e.callcommand(
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	195 b'rawstorefiledata',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	196 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	197 b'files': [b'changelog', b'manifestlog'],
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	198 },
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	199 ).result()
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	200
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	201 # First object is a summary of files data that follows.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	202 overall = next(objs)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	203
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	204 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	205 _(b'clone'), total=overall[b'totalsize'], unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	206 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	207 with progress:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	208 progress.update(0)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	209
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	210 # Next are pairs of file metadata, data.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	211 while True:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	212 try:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	213 filemeta = next(objs)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	214 except StopIteration:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	215 break
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	216
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	217 for k in (b'location', b'path', b'size'):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	218 if k not in filemeta:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	219 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	220 _(b'remote file data missing key: %s') % k
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	221 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	222
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	223 if filemeta[b'location'] == b'store':
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	224 vfs = repo.svfs
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	225 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	226 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43106 diff changeset	227 _(b'invalid location for raw file data: %s')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	228 % filemeta[b'location']
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	229 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	230
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	231 bytesremaining = filemeta[b'size']
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	232
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	233 with vfs.open(filemeta[b'path'], b'wb') as fh:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	234 while True:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	235 try:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	236 chunk = next(objs)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	237 except StopIteration:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	238 break
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	239
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	240 bytesremaining -= len(chunk)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	241
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	242 if bytesremaining < 0:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	243 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	244 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	245 b'received invalid number of bytes for file '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	246 b'data; expected %d, got extra'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	247 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	248 % filemeta[b'size']
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	249 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	250
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	251 progress.increment(step=len(chunk))
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	252 fh.write(chunk)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	253
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	254 try:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	255 if chunk.islast:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	256 break
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	257 except AttributeError:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	258 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	259 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	260 b'did not receive indefinite length bytestring '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	261 b'for file data'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	262 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	263 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	264
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	265 if bytesremaining:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	266 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	267 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	268 b'received invalid number of bytes for'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	269 b'file data; expected %d got %d'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	270 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	271 % (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	272 filemeta[b'size'],
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	273 filemeta[b'size'] - bytesremaining,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	274 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	275 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	276
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	277
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	278 def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	279 """Determine which changesets need to be pulled."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	280
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	281 if heads:
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	282 knownnode = repo.changelog.hasnode
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	283 if all(knownnode(head) for head in heads):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	284 return heads, False, heads
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	285
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	286 # TODO wire protocol version 2 is capable of more efficient discovery
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	287 # than setdiscovery. Consider implementing something better.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	288 common, fetch, remoteheads = setdiscovery.findcommonheads(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	289 repo.ui, repo, remote, abortwhenunrelated=abortwhenunrelated
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	290 )
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	291
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	292 common = set(common)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	293 remoteheads = set(remoteheads)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	294
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	295 # If a remote head is filtered locally, put it back in the common set.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296 # See the comment in exchange._pulldiscoverychangegroup() for more.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298 if fetch and remoteheads:
43594 a166fadf5c3b index: use `index.has_node` in `exchangev2._pullchangesetdiscovery` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 43117 diff changeset	299 has_node = repo.unfiltered().changelog.index.has_node
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300
43594 a166fadf5c3b index: use `index.has_node` in `exchangev2._pullchangesetdiscovery` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 43117 diff changeset	301 common \|= {head for head in remoteheads if has_node(head)}
39645 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	302
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	303 if set(remoteheads).issubset(common):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	304 fetch = []
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	305
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	306 common.discard(nullid)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	307
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	308 return common, fetch, remoteheads
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	309
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	310
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	311 def _fetchchangesets(repo, tr, remote, common, fetch, remoteheads):
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	312 # TODO consider adding a step here where we obtain the DAG shape first
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	313 # (or ask the server to slice changesets into chunks for us) so that
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	314 # we can perform multiple fetches in batches. This will facilitate
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	315 # resuming interrupted clones, higher server-side cache hit rates due
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	316 # to smaller segments, etc.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	317 with remote.commandexecutor() as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	318 objs = e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	319 b'changesetdata',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	320 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	321 b'revisions': [
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	322 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	323 b'type': b'changesetdagrange',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	324 b'roots': sorted(common),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	325 b'heads': sorted(remoteheads),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	326 }
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	327 ],
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	328 b'fields': {b'bookmarks', b'parents', b'phase', b'revision'},
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	329 },
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	330 ).result()
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	331
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	332 # The context manager waits on all response data when exiting. So
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	333 # we need to remain in the context manager in order to stream data.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	334 return _processchangesetdata(repo, tr, objs)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	335
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	336
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	337 def _processchangesetdata(repo, tr, objs):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	338 repo.hook(b'prechangegroup', throw=True, **pycompat.strkwargs(tr.hookargs))
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	339
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	340 urepo = repo.unfiltered()
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	341 cl = urepo.changelog
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	342
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	343 cl.delayupdate(tr)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	344
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	345 # The first emitted object is a header describing the data that
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	346 # follows.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	347 meta = next(objs)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	348
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	349 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	350 _(b'changesets'), unit=_(b'chunks'), total=meta.get(b'totalitems')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	351 )
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	352
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	353 manifestnodes = {}
45811 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	354 added = []
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	355
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	356 def linkrev(node):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	357 repo.ui.debug(b'add changeset %s\n' % short(node))
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	358 # Linkrev for changelog is always self.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	359 return len(cl)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	360
45811 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	361 def ondupchangeset(cl, node):
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	362 added.append(node)
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	363
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	364 def onchangeset(cl, node):
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	365 progress.increment()
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	366
46432 009130064813 exchangev2: avoid second look-up by node Joerg Sonnenberger <joerg@bec.de> parents: 45957 diff changeset	367 rev = cl.rev(node)
009130064813 exchangev2: avoid second look-up by node Joerg Sonnenberger <joerg@bec.de> parents: 45957 diff changeset	368 revision = cl.changelogrevision(rev)
45811 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	369 added.append(node)
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	370
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	371 # We need to preserve the mapping of changelog revision to node
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	372 # so we can set the linkrev accordingly when manifests are added.
46432 009130064813 exchangev2: avoid second look-up by node Joerg Sonnenberger <joerg@bec.de> parents: 45957 diff changeset	373 manifestnodes[rev] = revision.manifest
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	374
46443 0903d6b9b1df repository: introduce register_changeset callback Joerg Sonnenberger <joerg@bec.de> parents: 46432 diff changeset	375 repo.register_changeset(rev, revision)
0903d6b9b1df repository: introduce register_changeset callback Joerg Sonnenberger <joerg@bec.de> parents: 46432 diff changeset	376
45117 b1e51ef4e536 phases: sparsify phase lists Joerg Sonnenberger <joerg@bec.de> parents: 43594 diff changeset	377 nodesbyphase = {phase: set() for phase in phases.phasenames.values()}
39651 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39649 diff changeset	378 remotebookmarks = {}
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	379
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	380 # addgroup() expects a 7-tuple describing revisions. This normalizes
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	381 # the wire data to that format.
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	382 #
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	383 # This loop also aggregates non-revision metadata, such as phase
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	384 # data.
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	385 def iterrevisions():
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	386 for cset in objs:
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	387 node = cset[b'node']
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	388
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	389 if b'phase' in cset:
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	390 nodesbyphase[cset[b'phase']].add(node)
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	391
39651 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39649 diff changeset	392 for mark in cset.get(b'bookmarks', []):
349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39649 diff changeset	393 remotebookmarks[mark] = node
349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39649 diff changeset	394
39652 399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields Gregory Szorc <gregory.szorc@gmail.com> parents: 39651 diff changeset	395 # TODO add mechanism for extensions to examine records so they
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields Gregory Szorc <gregory.szorc@gmail.com> parents: 39651 diff changeset	396 # can siphon off custom data fields.
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields Gregory Szorc <gregory.szorc@gmail.com> parents: 39651 diff changeset	397
39819 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	398 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	399
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	400 for field, size in cset.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	401 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	402
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	403 # Some entries might only be metadata only updates.
39819 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	404 if b'revision' not in extrafields:
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	405 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	406
39819 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	407 data = extrafields[b'revision']
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	408
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	409 yield (
39649 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39647 diff changeset	410 node,
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	411 cset[b'parents'][0],
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	412 cset[b'parents'][1],
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	413 # Linknode is always itself for changesets.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	414 cset[b'node'],
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	415 # We always send full revisions. So delta base is not set.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	416 nullid,
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	417 mdiff.trivialdiffheader(len(data)) + data,
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	418 # Flags not yet supported.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	419 0,
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	420 )
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	421
45811 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	422 cl.addgroup(
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	423 iterrevisions(),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	424 linkrev,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	425 weakref.proxy(tr),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	426 addrevisioncb=onchangeset,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	427 duplicaterevisioncb=ondupchangeset,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	428 )
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	429
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	430 progress.complete()
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	431
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	432 return {
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	433 b'added': added,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	434 b'nodesbyphase': nodesbyphase,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	435 b'bookmarks': remotebookmarks,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	436 b'manifestnodes': manifestnodes,
39647 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39645 diff changeset	437 }
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	438
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	439
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	440 def _fetchmanifests(repo, tr, remote, manifestnodes):
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	441 rootmanifest = repo.manifestlog.getstorage(b'')
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	442
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	443 # Some manifests can be shared between changesets. Filter out revisions
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	444 # we already know about.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	445 fetchnodes = []
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	446 linkrevs = {}
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	447 seen = set()
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	448
43106 d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer Gregory Szorc <gregory.szorc@gmail.com> parents: 43104 diff changeset	449 for clrev, node in sorted(pycompat.iteritems(manifestnodes)):
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	450 if node in seen:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	451 continue
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	452
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	453 try:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	454 rootmanifest.rev(node)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	455 except error.LookupError:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	456 fetchnodes.append(node)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	457 linkrevs[node] = clrev
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	458
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	459 seen.add(node)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	460
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	461 # TODO handle tree manifests
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	462
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	463 # addgroup() expects 7-tuple describing revisions. This normalizes
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	464 # the wire data to that format.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	465 def iterrevisions(objs, progress):
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	466 for manifest in objs:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	467 node = manifest[b'node']
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	468
39819 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	469 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	470
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	471 for field, size in manifest.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	472 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	473
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	474 if b'delta' in extrafields:
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	475 basenode = manifest[b'deltabasenode']
39819 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	476 delta = extrafields[b'delta']
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	477 elif b'revision' in extrafields:
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	478 basenode = nullid
39819 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	479 revision = extrafields[b'revision']
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	480 delta = mdiff.trivialdiffheader(len(revision)) + revision
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	481 else:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	482 continue
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	483
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	484 yield (
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	485 node,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	486 manifest[b'parents'][0],
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	487 manifest[b'parents'][1],
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	488 # The value passed in is passed to the lookup function passed
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	489 # to addgroup(). We already have a map of manifest node to
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	490 # changelog revision number. So we just pass in the
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	491 # manifest node here and use linkrevs.__getitem__ as the
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	492 # resolution function.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	493 node,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	494 basenode,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	495 delta,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	496 # Flags not yet supported.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	497 0,
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	498 )
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	499
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	500 progress.increment()
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	501
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	502 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	503 _(b'manifests'), unit=_(b'chunks'), total=len(fetchnodes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	504 )
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	505
40173 b797150a1ab9 exchangev2: honor server advertised manifestdata recommended batch size Gregory Szorc <gregory.szorc@gmail.com> parents: 40036 diff changeset	506 commandmeta = remote.apidescriptor[b'commands'][b'manifestdata']
b797150a1ab9 exchangev2: honor server advertised manifestdata recommended batch size Gregory Szorc <gregory.szorc@gmail.com> parents: 40036 diff changeset	507 batchsize = commandmeta.get(b'recommendedbatchsize', 10000)
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	508 # TODO make size configurable on client?
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	509
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	510 # We send commands 1 at a time to the remote. This is not the most
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	511 # efficient because we incur a round trip at the end of each batch.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	512 # However, the existing frame-based reactor keeps consuming server
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	513 # data in the background. And this results in response data buffering
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	514 # in memory. This can consume gigabytes of memory.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	515 # TODO send multiple commands in a request once background buffering
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	516 # issues are resolved.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	517
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	518 added = []
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	519
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	520 for i in pycompat.xrange(0, len(fetchnodes), batchsize):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	521 batch = [node for node in fetchnodes[i : i + batchsize]]
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	522 if not batch:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	523 continue
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	524
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	525 with remote.commandexecutor() as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	526 objs = e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	527 b'manifestdata',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	528 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	529 b'tree': b'',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	530 b'nodes': batch,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	531 b'fields': {b'parents', b'revision'},
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	532 b'haveparents': True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	533 },
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	534 ).result()
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	535
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	536 # Chomp off header object.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	537 next(objs)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	538
45811 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	539 def onchangeset(cl, node):
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	540 added.append(node)
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	541
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	542 rootmanifest.addgroup(
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	543 iterrevisions(objs, progress),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	544 linkrevs.__getitem__,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	545 weakref.proxy(tr),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	546 addrevisioncb=onchangeset,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	547 duplicaterevisioncb=onchangeset,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	548 )
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	549
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	550 progress.complete()
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	551
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	552 return {
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	553 b'added': added,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	554 b'linkrevs': linkrevs,
39654 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39652 diff changeset	555 }
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	556
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	557
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	558 def _derivefilesfrommanifests(repo, matcher, manifestnodes):
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	559 """Determine what file nodes are relevant given a set of manifest nodes.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	560
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	561 Returns a dict mapping file paths to dicts of file node to first manifest
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	562 node.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	563 """
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	564 ml = repo.manifestlog
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	565 fnodes = collections.defaultdict(dict)
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	566
40036 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	567 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	568 _(b'scanning manifests'), total=len(manifestnodes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	569 )
40036 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	570
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	571 with progress:
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	572 for manifestnode in manifestnodes:
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	573 m = ml.get(b'', manifestnode)
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	574
40036 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	575 # TODO this will pull in unwanted nodes because it takes the storage
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	576 # delta into consideration. What we really want is something that
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	577 # takes the delta between the manifest's parents. And ideally we
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	578 # would ignore file nodes that are known locally. For now, ignore
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	579 # both these limitations. This will result in incremental fetches
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	580 # requesting data we already have. So this is far from ideal.
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	581 md = m.readfast()
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	582
40036 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	583 for path, fnode in md.items():
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	584 if matcher(path):
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	585 fnodes[path].setdefault(fnode, manifestnode)
40036 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	586
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39819 diff changeset	587 progress.increment()
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	588
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	589 return fnodes
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	590
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	591
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	592 def _fetchfiles(repo, tr, remote, fnodes, linkrevs):
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	593 """Fetch file data from explicit file revisions."""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	594
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	595 def iterrevisions(objs, progress):
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	596 for filerevision in objs:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	597 node = filerevision[b'node']
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	598
39819 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	599 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	600
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	601 for field, size in filerevision.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	602 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	603
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	604 if b'delta' in extrafields:
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	605 basenode = filerevision[b'deltabasenode']
39819 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	606 delta = extrafields[b'delta']
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	607 elif b'revision' in extrafields:
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	608 basenode = nullid
39819 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39657 diff changeset	609 revision = extrafields[b'revision']
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	610 delta = mdiff.trivialdiffheader(len(revision)) + revision
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	611 else:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	612 continue
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	613
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	614 yield (
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	615 node,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	616 filerevision[b'parents'][0],
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	617 filerevision[b'parents'][1],
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	618 node,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	619 basenode,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	620 delta,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	621 # Flags not yet supported.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	622 0,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	623 )
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	624
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	625 progress.increment()
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	626
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	627 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	628 _(b'files'),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	629 unit=_(b'chunks'),
43104 74802979dd9d py3: define and use pycompat.itervalues() Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	630 total=sum(len(v) for v in pycompat.itervalues(fnodes)),
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	631 )
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	632
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	633 # TODO make batch size configurable
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	634 batchsize = 10000
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	635 fnodeslist = [x for x in sorted(fnodes.items())]
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	636
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	637 for i in pycompat.xrange(0, len(fnodeslist), batchsize):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	638 batch = [x for x in fnodeslist[i : i + batchsize]]
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	639 if not batch:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	640 continue
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	641
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	642 with remote.commandexecutor() as e:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	643 fs = []
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	644 locallinkrevs = {}
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	645
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	646 for path, nodes in batch:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	647 fs.append(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	648 (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	649 path,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	650 e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	651 b'filedata',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	652 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	653 b'path': path,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	654 b'nodes': sorted(nodes),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	655 b'fields': {b'parents', b'revision'},
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	656 b'haveparents': True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	657 },
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	658 ),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	659 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	660 )
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	661
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	662 locallinkrevs[path] = {
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	663 node: linkrevs[manifestnode]
43106 d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer Gregory Szorc <gregory.szorc@gmail.com> parents: 43104 diff changeset	664 for node, manifestnode in pycompat.iteritems(nodes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	665 }
39656 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	666
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	667 for path, f in fs:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	668 objs = f.result()
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	669
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	670 # Chomp off header objects.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	671 next(objs)
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	672
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	673 store = repo.file(path)
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	674 store.addgroup(
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	675 iterrevisions(objs, progress),
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39654 diff changeset	676 locallinkrevs[path].__getitem__,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	677 weakref.proxy(tr),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	678 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	679
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	680
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	681 def _fetchfilesfromcsets(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	682 repo, tr, remote, pathfilter, fnodes, csets, manlinkrevs, shallow=False
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	683 ):
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	684 """Fetch file data from explicit changeset revisions."""
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	685
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	686 def iterrevisions(objs, remaining, progress):
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	687 while remaining:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	688 filerevision = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	689
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	690 node = filerevision[b'node']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	691
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	692 extrafields = {}
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	693
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	694 for field, size in filerevision.get(b'fieldsfollowing', []):
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	695 extrafields[field] = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	696
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	697 if b'delta' in extrafields:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	698 basenode = filerevision[b'deltabasenode']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	699 delta = extrafields[b'delta']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	700 elif b'revision' in extrafields:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	701 basenode = nullid
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	702 revision = extrafields[b'revision']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	703 delta = mdiff.trivialdiffheader(len(revision)) + revision
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	704 else:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	705 continue
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	706
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	707 if b'linknode' in filerevision:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	708 linknode = filerevision[b'linknode']
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	709 else:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	710 linknode = node
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	711
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	712 yield (
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	713 node,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	714 filerevision[b'parents'][0],
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	715 filerevision[b'parents'][1],
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	716 linknode,
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	717 basenode,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	718 delta,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	719 # Flags not yet supported.
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	720 0,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	721 )
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	722
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	723 progress.increment()
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	724 remaining -= 1
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	725
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	726 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	727 _(b'files'),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	728 unit=_(b'chunks'),
43104 74802979dd9d py3: define and use pycompat.itervalues() Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	729 total=sum(len(v) for v in pycompat.itervalues(fnodes)),
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	730 )
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	731
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	732 commandmeta = remote.apidescriptor[b'commands'][b'filesdata']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	733 batchsize = commandmeta.get(b'recommendedbatchsize', 50000)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	734
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	735 shallowfiles = repository.REPO_FEATURE_SHALLOW_FILE_STORAGE in repo.features
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	736 fields = {b'parents', b'revision'}
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	737 clrev = repo.changelog.rev
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	738
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	739 # There are no guarantees that we'll have ancestor revisions if
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	740 # a) this repo has shallow file storage b) shallow data fetching is enabled.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	741 # Force remote to not delta against possibly unknown revisions when these
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	742 # conditions hold.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	743 haveparents = not (shallowfiles or shallow)
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	744
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	745 # Similarly, we may not have calculated linkrevs for all incoming file
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	746 # revisions. Ask the remote to do work for us in this case.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	747 if not haveparents:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	748 fields.add(b'linknode')
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	749
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	750 for i in pycompat.xrange(0, len(csets), batchsize):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	751 batch = [x for x in csets[i : i + batchsize]]
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	752 if not batch:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	753 continue
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	754
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	755 with remote.commandexecutor() as e:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	756 args = {
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	757 b'revisions': [
45957 89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	758 {
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	759 b'type': b'changesetexplicit',
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	760 b'nodes': batch,
89a2afe31e82 formating: upgrade to black 20.8b1 Augie Fackler <raf@durin42.com> parents: 45813 diff changeset	761 }
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	762 ],
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	763 b'fields': fields,
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	764 b'haveparents': haveparents,
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	765 }
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	766
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	767 if pathfilter:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	768 args[b'pathfilter'] = pathfilter
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	769
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	770 objs = e.callcommand(b'filesdata', args).result()
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	771
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	772 # First object is an overall header.
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	773 overall = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	774
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	775 # We have overall['totalpaths'] segments.
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	776 for i in pycompat.xrange(overall[b'totalpaths']):
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	777 header = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	778
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	779 path = header[b'path']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	780 store = repo.file(path)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	781
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	782 linkrevs = {
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	783 fnode: manlinkrevs[mnode]
43106 d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer Gregory Szorc <gregory.szorc@gmail.com> parents: 43104 diff changeset	784 for fnode, mnode in pycompat.iteritems(fnodes[path])
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	785 }
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	786
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	787 def getlinkrev(node):
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	788 if node in linkrevs:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	789 return linkrevs[node]
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	790 else:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	791 return clrev(node)
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	792
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	793 store.addgroup(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	794 iterrevisions(objs, header[b'totalitems'], progress),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	795 getlinkrev,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	796 weakref.proxy(tr),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	797 maybemissingparents=shallow,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	798 )

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/exchangev2.py @ 46443:0903d6b9b1df