mercurial-scm/hg: mercurial/exchangev2.py annotate

annotate mercurial/exchangev2.py @ 45788:a5206e71c536

revlog: extend addgroup() with callback for duplicates The addgroup() interface currently doesn't allow the caller to keep track of duplicated nodes except by looking at the returned node list. Add an optional second callback for this purpose and change the return type to a boolean. This allows follow-up changes to use more efficient storage for the node list in places that are memory-sensitive. Differential Revision: https://phab.mercurial-scm.org/D9231

author	Joerg Sonnenberger <joerg@bec.de>
date	Sun, 18 Oct 2020 22:18:02 +0200
parents	b1e51ef4e536
children	5d65e04b6a80

rev	line source
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # exchangev2.py - repository exchange for wire protocol version 2
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	8 from __future__ import absolute_import
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	10 import collections
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	11 import weakref
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	12
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	13 from .i18n import _
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	14 from .node import (
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	15 nullid,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	16 short,
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	17 )
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	18 from . import (
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	19 bookmarks,
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	20 error,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	21 mdiff,
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	22 narrowspec,
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	23 phases,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	24 pycompat,
42813 268662aac075 interfaces: create a new folder for interfaces and move repository.py in it Pulkit Goyal <pulkit@yandex-team.ru> parents: 41217 diff changeset	25 setdiscovery,
268662aac075 interfaces: create a new folder for interfaces and move repository.py in it Pulkit Goyal <pulkit@yandex-team.ru> parents: 41217 diff changeset	26 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	27 from .interfaces import repository
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	28
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	30 def pull(pullop):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	31 """Pull using wire protocol version 2."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 repo = pullop.repo
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	33 remote = pullop.remote
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	34
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	35 usingrawchangelogandmanifest = _checkuserawstorefiledata(pullop)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	36
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	37 # If this is a clone and it was requested to perform a "stream clone",
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	38 # we obtain the raw files data from the remote then fall back to an
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	39 # incremental pull. This is somewhat hacky and is not nearly robust enough
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	40 # for long-term usage.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	41 if usingrawchangelogandmanifest:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	42 with repo.transaction(b'clone'):
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	43 _fetchrawstorefiles(repo, remote)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	44 repo.invalidate(clearfilecache=True)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	45
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	46 tr = pullop.trmanager.transaction()
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	47
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	48 # We don't use the repo's narrow matcher here because the patterns passed
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	49 # to exchange.pull() could be different.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	50 narrowmatcher = narrowspec.match(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	51 repo.root,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	52 # Empty maps to nevermatcher. So always
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	53 # set includes if missing.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	54 pullop.includepats or {b'path:.'},
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	55 pullop.excludepats,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	56 )
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	57
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	58 if pullop.includepats or pullop.excludepats:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	59 pathfilter = {}
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	60 if pullop.includepats:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	61 pathfilter[b'include'] = sorted(pullop.includepats)
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	62 if pullop.excludepats:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	63 pathfilter[b'exclude'] = sorted(pullop.excludepats)
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	64 else:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	65 pathfilter = None
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	66
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	67 # Figure out what needs to be fetched.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	68 common, fetch, remoteheads = _pullchangesetdiscovery(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	69 repo, remote, pullop.heads, abortwhenunrelated=pullop.force
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	70 )
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	71
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	72 # And fetch the data.
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	73 pullheads = pullop.heads or remoteheads
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	74 csetres = _fetchchangesets(repo, tr, remote, common, fetch, pullheads)
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	75
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	76 # New revisions are written to the changelog. But all other updates
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	77 # are deferred. Do those now.
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	78
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	79 # Ensure all new changesets are draft by default. If the repo is
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	80 # publishing, the phase will be adjusted by the loop below.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	81 if csetres[b'added']:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	82 phases.registernew(repo, tr, phases.draft, csetres[b'added'])
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	83
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	84 # And adjust the phase of all changesets accordingly.
45117 b1e51ef4e536 phases: sparsify phase lists Joerg Sonnenberger <joerg@bec.de> parents: 43546 diff changeset	85 for phasenumber, phase in phases.phasenames.items():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	86 if phase == b'secret' or not csetres[b'nodesbyphase'][phase]:
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	87 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	88
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	89 phases.advanceboundary(
45117 b1e51ef4e536 phases: sparsify phase lists Joerg Sonnenberger <joerg@bec.de> parents: 43546 diff changeset	90 repo, tr, phasenumber, csetres[b'nodesbyphase'][phase],
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	91 )
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	92
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	93 # Write bookmark updates.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	94 bookmarks.updatefromremote(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	95 repo.ui,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	96 repo,
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	97 csetres[b'bookmarks'],
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	98 remote.url(),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	99 pullop.gettransaction,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	100 explicit=pullop.explicitbookmarks,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	101 )
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	102
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	103 manres = _fetchmanifests(repo, tr, remote, csetres[b'manifestnodes'])
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	104
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	105 # We don't properly support shallow changeset and manifest yet. So we apply
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	106 # depth limiting locally.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	107 if pullop.depth:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	108 relevantcsetnodes = set()
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	109 clnode = repo.changelog.node
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	110
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	111 for rev in repo.revs(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	112 b'ancestors(%ln, %s)', pullheads, pullop.depth - 1
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	113 ):
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	114 relevantcsetnodes.add(clnode(rev))
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	115
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	116 csetrelevantfilter = lambda n: n in relevantcsetnodes
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	117
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	118 else:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	119 csetrelevantfilter = lambda n: True
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	120
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	121 # If obtaining the raw store files, we need to scan the full repo to
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	122 # derive all the changesets, manifests, and linkrevs.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	123 if usingrawchangelogandmanifest:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	124 csetsforfiles = []
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	125 mnodesforfiles = []
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	126 manifestlinkrevs = {}
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	127
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	128 for rev in repo:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	129 ctx = repo[rev]
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	130 node = ctx.node()
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	131
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	132 if not csetrelevantfilter(node):
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	133 continue
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	134
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	135 mnode = ctx.manifestnode()
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	136
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	137 csetsforfiles.append(node)
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	138 mnodesforfiles.append(mnode)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	139 manifestlinkrevs[mnode] = rev
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	140
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	141 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	142 csetsforfiles = [n for n in csetres[b'added'] if csetrelevantfilter(n)]
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	143 mnodesforfiles = manres[b'added']
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	144 manifestlinkrevs = manres[b'linkrevs']
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	145
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	146 # Find all file nodes referenced by added manifests and fetch those
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	147 # revisions.
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	148 fnodes = _derivefilesfrommanifests(repo, narrowmatcher, mnodesforfiles)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	149 _fetchfilesfromcsets(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	150 repo,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	151 tr,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	152 remote,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	153 pathfilter,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	154 fnodes,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	155 csetsforfiles,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	156 manifestlinkrevs,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	157 shallow=bool(pullop.depth),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	158 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	159
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	160
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	161 def _checkuserawstorefiledata(pullop):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	162 """Check whether we should use rawstorefiledata command to retrieve data."""
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	163
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	164 repo = pullop.repo
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	165 remote = pullop.remote
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	166
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	167 # Command to obtain raw store data isn't available.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	168 if b'rawstorefiledata' not in remote.apidescriptor[b'commands']:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	169 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	170
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	171 # Only honor if user requested stream clone operation.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	172 if not pullop.streamclonerequested:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	173 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	174
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	175 # Only works on empty repos.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	176 if len(repo):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	177 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	178
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	179 # TODO This is super hacky. There needs to be a storage API for this. We
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	180 # also need to check for compatibility with the remote.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	181 if b'revlogv1' not in repo.requirements:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	182 return False
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	183
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	184 return True
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	185
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	186
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	187 def _fetchrawstorefiles(repo, remote):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	188 with remote.commandexecutor() as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	189 objs = e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	190 b'rawstorefiledata', {b'files': [b'changelog', b'manifestlog'],}
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	191 ).result()
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	192
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	193 # First object is a summary of files data that follows.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	194 overall = next(objs)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	195
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	196 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	197 _(b'clone'), total=overall[b'totalsize'], unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	198 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	199 with progress:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	200 progress.update(0)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	201
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	202 # Next are pairs of file metadata, data.
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	203 while True:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	204 try:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	205 filemeta = next(objs)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	206 except StopIteration:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	207 break
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	208
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	209 for k in (b'location', b'path', b'size'):
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	210 if k not in filemeta:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	211 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	212 _(b'remote file data missing key: %s') % k
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	213 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	214
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	215 if filemeta[b'location'] == b'store':
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	216 vfs = repo.svfs
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	217 else:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	218 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43106 diff changeset	219 _(b'invalid location for raw file data: %s')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	220 % filemeta[b'location']
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	221 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	222
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	223 bytesremaining = filemeta[b'size']
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	224
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	225 with vfs.open(filemeta[b'path'], b'wb') as fh:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	226 while True:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	227 try:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	228 chunk = next(objs)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	229 except StopIteration:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	230 break
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	231
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	232 bytesremaining -= len(chunk)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	233
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	234 if bytesremaining < 0:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	235 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	236 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	237 b'received invalid number of bytes for file '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	238 b'data; expected %d, got extra'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	239 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	240 % filemeta[b'size']
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	241 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	242
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	243 progress.increment(step=len(chunk))
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	244 fh.write(chunk)
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	245
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	246 try:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	247 if chunk.islast:
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	248 break
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	249 except AttributeError:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	250 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	251 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	252 b'did not receive indefinite length bytestring '
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	253 b'for file data'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	254 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	255 )
40330 00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	256
00a4cd368e3f exchangev2: support for calling rawstorefiledata to retrieve raw files Gregory Szorc <gregory.szorc@gmail.com> parents: 40327 diff changeset	257 if bytesremaining:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	258 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	259 _(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	260 b'received invalid number of bytes for'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	261 b'file data; expected %d got %d'
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	262 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	263 % (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	264 filemeta[b'size'],
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	265 filemeta[b'size'] - bytesremaining,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	266 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	267 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	268
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	269
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	270 def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	271 """Determine which changesets need to be pulled."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	272
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	273 if heads:
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	274 knownnode = repo.changelog.hasnode
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	275 if all(knownnode(head) for head in heads):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	276 return heads, False, heads
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	277
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	278 # TODO wire protocol version 2 is capable of more efficient discovery
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	279 # than setdiscovery. Consider implementing something better.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	280 common, fetch, remoteheads = setdiscovery.findcommonheads(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	281 repo.ui, repo, remote, abortwhenunrelated=abortwhenunrelated
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	282 )
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	283
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	284 common = set(common)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	285 remoteheads = set(remoteheads)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	286
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	287 # If a remote head is filtered locally, put it back in the common set.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	288 # See the comment in exchange._pulldiscoverychangegroup() for more.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	289
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	290 if fetch and remoteheads:
43546 a166fadf5c3b index: use `index.has_node` in `exchangev2._pullchangesetdiscovery` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 43117 diff changeset	291 has_node = repo.unfiltered().changelog.index.has_node
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	292
43546 a166fadf5c3b index: use `index.has_node` in `exchangev2._pullchangesetdiscovery` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 43117 diff changeset	293 common \|= {head for head in remoteheads if has_node(head)}
39629 a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	294
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	295 if set(remoteheads).issubset(common):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	296 fetch = []
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	297
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	298 common.discard(nullid)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	299
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2 Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	300 return common, fetch, remoteheads
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	301
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	302
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	303 def _fetchchangesets(repo, tr, remote, common, fetch, remoteheads):
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	304 # TODO consider adding a step here where we obtain the DAG shape first
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	305 # (or ask the server to slice changesets into chunks for us) so that
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	306 # we can perform multiple fetches in batches. This will facilitate
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	307 # resuming interrupted clones, higher server-side cache hit rates due
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	308 # to smaller segments, etc.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	309 with remote.commandexecutor() as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	310 objs = e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	311 b'changesetdata',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	312 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	313 b'revisions': [
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	314 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	315 b'type': b'changesetdagrange',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	316 b'roots': sorted(common),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	317 b'heads': sorted(remoteheads),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	318 }
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	319 ],
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	320 b'fields': {b'bookmarks', b'parents', b'phase', b'revision'},
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	321 },
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	322 ).result()
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	323
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	324 # The context manager waits on all response data when exiting. So
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	325 # we need to remain in the context manager in order to stream data.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	326 return _processchangesetdata(repo, tr, objs)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	327
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	328
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	329 def _processchangesetdata(repo, tr, objs):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	330 repo.hook(b'prechangegroup', throw=True, **pycompat.strkwargs(tr.hookargs))
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	331
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	332 urepo = repo.unfiltered()
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	333 cl = urepo.changelog
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	334
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	335 cl.delayupdate(tr)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	336
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	337 # The first emitted object is a header describing the data that
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	338 # follows.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	339 meta = next(objs)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	340
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	341 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	342 _(b'changesets'), unit=_(b'chunks'), total=meta.get(b'totalitems')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	343 )
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	344
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	345 manifestnodes = {}
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	346 added = []
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	347
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	348 def linkrev(node):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	349 repo.ui.debug(b'add changeset %s\n' % short(node))
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	350 # Linkrev for changelog is always self.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	351 return len(cl)
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	352
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	353 def ondupchangeset(cl, node):
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	354 added.append(node)
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	355
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	356 def onchangeset(cl, node):
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	357 progress.increment()
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	358
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	359 revision = cl.changelogrevision(node)
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	360 added.append(node)
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	361
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	362 # We need to preserve the mapping of changelog revision to node
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	363 # so we can set the linkrev accordingly when manifests are added.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	364 manifestnodes[cl.rev(node)] = revision.manifest
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	365
45117 b1e51ef4e536 phases: sparsify phase lists Joerg Sonnenberger <joerg@bec.de> parents: 43546 diff changeset	366 nodesbyphase = {phase: set() for phase in phases.phasenames.values()}
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	367 remotebookmarks = {}
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	368
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	369 # addgroup() expects a 7-tuple describing revisions. This normalizes
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	370 # the wire data to that format.
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	371 #
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	372 # This loop also aggregates non-revision metadata, such as phase
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	373 # data.
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	374 def iterrevisions():
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	375 for cset in objs:
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	376 node = cset[b'node']
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	377
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	378 if b'phase' in cset:
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	379 nodesbyphase[cset[b'phase']].add(node)
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	380
39635 349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	381 for mark in cset.get(b'bookmarks', []):
349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	382 remotebookmarks[mark] = node
349482d726ee exchangev2: fetch and apply bookmarks Gregory Szorc <gregory.szorc@gmail.com> parents: 39633 diff changeset	383
39636 399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields Gregory Szorc <gregory.szorc@gmail.com> parents: 39635 diff changeset	384 # TODO add mechanism for extensions to examine records so they
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields Gregory Szorc <gregory.szorc@gmail.com> parents: 39635 diff changeset	385 # can siphon off custom data fields.
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields Gregory Szorc <gregory.szorc@gmail.com> parents: 39635 diff changeset	386
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	387 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	388
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	389 for field, size in cset.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	390 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	391
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	392 # Some entries might only be metadata only updates.
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	393 if b'revision' not in extrafields:
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	394 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	395
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	396 data = extrafields[b'revision']
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	397
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	398 yield (
39633 ff2de4f2eb3c exchangev2: fetch and apply phases data Gregory Szorc <gregory.szorc@gmail.com> parents: 39631 diff changeset	399 node,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	400 cset[b'parents'][0],
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	401 cset[b'parents'][1],
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	402 # Linknode is always itself for changesets.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	403 cset[b'node'],
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	404 # We always send full revisions. So delta base is not set.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	405 nullid,
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	406 mdiff.trivialdiffheader(len(data)) + data,
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	407 # Flags not yet supported.
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	408 0,
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	409 )
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	410
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	411 cl.addgroup(
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	412 iterrevisions(),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	413 linkrev,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	414 weakref.proxy(tr),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	415 addrevisioncb=onchangeset,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	416 duplicaterevisioncb=ondupchangeset,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	417 )
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	418
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	419 progress.complete()
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	420
b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	421 return {
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	422 b'added': added,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	423 b'nodesbyphase': nodesbyphase,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	424 b'bookmarks': remotebookmarks,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	425 b'manifestnodes': manifestnodes,
39631 b9e453d683a1 exchangev2: fetch changeset revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39629 diff changeset	426 }
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	427
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	428
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	429 def _fetchmanifests(repo, tr, remote, manifestnodes):
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	430 rootmanifest = repo.manifestlog.getstorage(b'')
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	431
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	432 # Some manifests can be shared between changesets. Filter out revisions
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	433 # we already know about.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	434 fetchnodes = []
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	435 linkrevs = {}
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	436 seen = set()
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	437
43106 d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer Gregory Szorc <gregory.szorc@gmail.com> parents: 43104 diff changeset	438 for clrev, node in sorted(pycompat.iteritems(manifestnodes)):
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	439 if node in seen:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	440 continue
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	441
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	442 try:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	443 rootmanifest.rev(node)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	444 except error.LookupError:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	445 fetchnodes.append(node)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	446 linkrevs[node] = clrev
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	447
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	448 seen.add(node)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	449
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	450 # TODO handle tree manifests
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	451
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	452 # addgroup() expects 7-tuple describing revisions. This normalizes
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	453 # the wire data to that format.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	454 def iterrevisions(objs, progress):
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	455 for manifest in objs:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	456 node = manifest[b'node']
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	457
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	458 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	459
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	460 for field, size in manifest.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	461 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	462
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	463 if b'delta' in extrafields:
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	464 basenode = manifest[b'deltabasenode']
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	465 delta = extrafields[b'delta']
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	466 elif b'revision' in extrafields:
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	467 basenode = nullid
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	468 revision = extrafields[b'revision']
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	469 delta = mdiff.trivialdiffheader(len(revision)) + revision
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	470 else:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	471 continue
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	472
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	473 yield (
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	474 node,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	475 manifest[b'parents'][0],
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	476 manifest[b'parents'][1],
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	477 # The value passed in is passed to the lookup function passed
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	478 # to addgroup(). We already have a map of manifest node to
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	479 # changelog revision number. So we just pass in the
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	480 # manifest node here and use linkrevs.__getitem__ as the
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	481 # resolution function.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	482 node,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	483 basenode,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	484 delta,
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	485 # Flags not yet supported.
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	486 0,
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	487 )
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	488
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	489 progress.increment()
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	490
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	491 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	492 _(b'manifests'), unit=_(b'chunks'), total=len(fetchnodes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	493 )
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	494
40173 b797150a1ab9 exchangev2: honor server advertised manifestdata recommended batch size Gregory Szorc <gregory.szorc@gmail.com> parents: 40035 diff changeset	495 commandmeta = remote.apidescriptor[b'commands'][b'manifestdata']
b797150a1ab9 exchangev2: honor server advertised manifestdata recommended batch size Gregory Szorc <gregory.szorc@gmail.com> parents: 40035 diff changeset	496 batchsize = commandmeta.get(b'recommendedbatchsize', 10000)
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	497 # TODO make size configurable on client?
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	498
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	499 # We send commands 1 at a time to the remote. This is not the most
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	500 # efficient because we incur a round trip at the end of each batch.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	501 # However, the existing frame-based reactor keeps consuming server
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	502 # data in the background. And this results in response data buffering
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	503 # in memory. This can consume gigabytes of memory.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	504 # TODO send multiple commands in a request once background buffering
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	505 # issues are resolved.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	506
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	507 added = []
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	508
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	509 for i in pycompat.xrange(0, len(fetchnodes), batchsize):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	510 batch = [node for node in fetchnodes[i : i + batchsize]]
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	511 if not batch:
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	512 continue
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	513
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	514 with remote.commandexecutor() as e:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	515 objs = e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	516 b'manifestdata',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	517 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	518 b'tree': b'',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	519 b'nodes': batch,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	520 b'fields': {b'parents', b'revision'},
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	521 b'haveparents': True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	522 },
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	523 ).result()
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	524
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	525 # Chomp off header object.
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	526 next(objs)
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	527
45788 a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	528 def onchangeset(cl, node):
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	529 added.append(node)
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	530
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	531 rootmanifest.addgroup(
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	532 iterrevisions(objs, progress),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	533 linkrevs.__getitem__,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	534 weakref.proxy(tr),
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	535 addrevisioncb=onchangeset,
a5206e71c536 revlog: extend addgroup() with callback for duplicates Joerg Sonnenberger <joerg@bec.de> parents: 45117 diff changeset	536 duplicaterevisioncb=onchangeset,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	537 )
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	538
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	539 progress.complete()
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	540
d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	541 return {
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	542 b'added': added,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	543 b'linkrevs': linkrevs,
39638 d292328e0143 exchangev2: fetch manifest revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39636 diff changeset	544 }
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	545
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	546
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	547 def _derivefilesfrommanifests(repo, matcher, manifestnodes):
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	548 """Determine what file nodes are relevant given a set of manifest nodes.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	549
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	550 Returns a dict mapping file paths to dicts of file node to first manifest
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	551 node.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	552 """
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	553 ml = repo.manifestlog
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	554 fnodes = collections.defaultdict(dict)
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	555
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	556 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	557 _(b'scanning manifests'), total=len(manifestnodes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	558 )
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	559
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	560 with progress:
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	561 for manifestnode in manifestnodes:
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	562 m = ml.get(b'', manifestnode)
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	563
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	564 # TODO this will pull in unwanted nodes because it takes the storage
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	565 # delta into consideration. What we really want is something that
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	566 # takes the delta between the manifest's parents. And ideally we
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	567 # would ignore file nodes that are known locally. For now, ignore
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	568 # both these limitations. This will result in incremental fetches
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	569 # requesting data we already have. So this is far from ideal.
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	570 md = m.readfast()
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	571
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	572 for path, fnode in md.items():
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	573 if matcher(path):
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	574 fnodes[path].setdefault(fnode, manifestnode)
40035 7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	575
7a347d362a45 exchangev2: add progress bar around manifest scanning Gregory Szorc <gregory.szorc@gmail.com> parents: 39814 diff changeset	576 progress.increment()
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	577
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	578 return fnodes
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	579
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	580
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	581 def _fetchfiles(repo, tr, remote, fnodes, linkrevs):
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	582 """Fetch file data from explicit file revisions."""
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	583
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	584 def iterrevisions(objs, progress):
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	585 for filerevision in objs:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	586 node = filerevision[b'node']
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	587
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	588 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	589
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	590 for field, size in filerevision.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	591 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	592
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	593 if b'delta' in extrafields:
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	594 basenode = filerevision[b'deltabasenode']
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	595 delta = extrafields[b'delta']
d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	596 elif b'revision' in extrafields:
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	597 basenode = nullid
39814 d059cb669632 wireprotov2: allow multiple fields to follow revision maps Gregory Szorc <gregory.szorc@gmail.com> parents: 39641 diff changeset	598 revision = extrafields[b'revision']
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	599 delta = mdiff.trivialdiffheader(len(revision)) + revision
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	600 else:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	601 continue
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	602
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	603 yield (
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	604 node,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	605 filerevision[b'parents'][0],
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	606 filerevision[b'parents'][1],
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	607 node,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	608 basenode,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	609 delta,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	610 # Flags not yet supported.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	611 0,
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	612 )
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	613
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	614 progress.increment()
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	615
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	616 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	617 _(b'files'),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	618 unit=_(b'chunks'),
43104 74802979dd9d py3: define and use pycompat.itervalues() Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	619 total=sum(len(v) for v in pycompat.itervalues(fnodes)),
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	620 )
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	621
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	622 # TODO make batch size configurable
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	623 batchsize = 10000
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	624 fnodeslist = [x for x in sorted(fnodes.items())]
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	625
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	626 for i in pycompat.xrange(0, len(fnodeslist), batchsize):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	627 batch = [x for x in fnodeslist[i : i + batchsize]]
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	628 if not batch:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	629 continue
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	630
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	631 with remote.commandexecutor() as e:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	632 fs = []
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	633 locallinkrevs = {}
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	634
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	635 for path, nodes in batch:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	636 fs.append(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	637 (
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	638 path,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	639 e.callcommand(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	640 b'filedata',
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	641 {
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	642 b'path': path,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	643 b'nodes': sorted(nodes),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	644 b'fields': {b'parents', b'revision'},
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	645 b'haveparents': True,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	646 },
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	647 ),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	648 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	649 )
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	650
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	651 locallinkrevs[path] = {
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	652 node: linkrevs[manifestnode]
43106 d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer Gregory Szorc <gregory.szorc@gmail.com> parents: 43104 diff changeset	653 for node, manifestnode in pycompat.iteritems(nodes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	654 }
39640 039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	655
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	656 for path, f in fs:
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	657 objs = f.result()
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	658
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	659 # Chomp off header objects.
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	660 next(objs)
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	661
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	662 store = repo.file(path)
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	663 store.addgroup(
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	664 iterrevisions(objs, progress),
039bf1eddc2e exchangev2: fetch file revisions Gregory Szorc <gregory.szorc@gmail.com> parents: 39638 diff changeset	665 locallinkrevs[path].__getitem__,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	666 weakref.proxy(tr),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	667 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	668
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	669
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	670 def _fetchfilesfromcsets(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	671 repo, tr, remote, pathfilter, fnodes, csets, manlinkrevs, shallow=False
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	672 ):
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	673 """Fetch file data from explicit changeset revisions."""
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	674
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	675 def iterrevisions(objs, remaining, progress):
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	676 while remaining:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	677 filerevision = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	678
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	679 node = filerevision[b'node']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	680
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	681 extrafields = {}
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	682
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	683 for field, size in filerevision.get(b'fieldsfollowing', []):
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	684 extrafields[field] = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	685
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	686 if b'delta' in extrafields:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	687 basenode = filerevision[b'deltabasenode']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	688 delta = extrafields[b'delta']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	689 elif b'revision' in extrafields:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	690 basenode = nullid
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	691 revision = extrafields[b'revision']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	692 delta = mdiff.trivialdiffheader(len(revision)) + revision
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	693 else:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	694 continue
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	695
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	696 if b'linknode' in filerevision:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	697 linknode = filerevision[b'linknode']
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	698 else:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	699 linknode = node
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	700
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	701 yield (
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	702 node,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	703 filerevision[b'parents'][0],
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	704 filerevision[b'parents'][1],
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	705 linknode,
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	706 basenode,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	707 delta,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	708 # Flags not yet supported.
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	709 0,
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	710 )
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	711
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	712 progress.increment()
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	713 remaining -= 1
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	714
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	715 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	716 _(b'files'),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	717 unit=_(b'chunks'),
43104 74802979dd9d py3: define and use pycompat.itervalues() Gregory Szorc <gregory.szorc@gmail.com> parents: 43077 diff changeset	718 total=sum(len(v) for v in pycompat.itervalues(fnodes)),
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	719 )
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	720
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	721 commandmeta = remote.apidescriptor[b'commands'][b'filesdata']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	722 batchsize = commandmeta.get(b'recommendedbatchsize', 50000)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	723
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	724 shallowfiles = repository.REPO_FEATURE_SHALLOW_FILE_STORAGE in repo.features
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	725 fields = {b'parents', b'revision'}
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	726 clrev = repo.changelog.rev
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	727
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	728 # There are no guarantees that we'll have ancestor revisions if
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	729 # a) this repo has shallow file storage b) shallow data fetching is enabled.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	730 # Force remote to not delta against possibly unknown revisions when these
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	731 # conditions hold.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	732 haveparents = not (shallowfiles or shallow)
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	733
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	734 # Similarly, we may not have calculated linkrevs for all incoming file
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	735 # revisions. Ask the remote to do work for us in this case.
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	736 if not haveparents:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	737 fields.add(b'linknode')
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	738
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	739 for i in pycompat.xrange(0, len(csets), batchsize):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	740 batch = [x for x in csets[i : i + batchsize]]
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	741 if not batch:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	742 continue
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	743
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	744 with remote.commandexecutor() as e:
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	745 args = {
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	746 b'revisions': [
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	747 {b'type': b'changesetexplicit', b'nodes': batch,}
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	748 ],
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	749 b'fields': fields,
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	750 b'haveparents': haveparents,
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	751 }
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	752
40327 55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	753 if pathfilter:
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	754 args[b'pathfilter'] = pathfilter
55836a34f41b exchangev2: recognize narrow patterns when pulling Gregory Szorc <gregory.szorc@gmail.com> parents: 40179 diff changeset	755
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	756 objs = e.callcommand(b'filesdata', args).result()
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	757
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	758 # First object is an overall header.
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	759 overall = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	760
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	761 # We have overall['totalpaths'] segments.
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	762 for i in pycompat.xrange(overall[b'totalpaths']):
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	763 header = next(objs)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	764
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	765 path = header[b'path']
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	766 store = repo.file(path)
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	767
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	768 linkrevs = {
b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	769 fnode: manlinkrevs[mnode]
43106 d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer Gregory Szorc <gregory.szorc@gmail.com> parents: 43104 diff changeset	770 for fnode, mnode in pycompat.iteritems(fnodes[path])
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	771 }
40179 b843356d4ae1 exchangev2: use filesdata Gregory Szorc <gregory.szorc@gmail.com> parents: 40176 diff changeset	772
40393 229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	773 def getlinkrev(node):
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	774 if node in linkrevs:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	775 return linkrevs[node]
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	776 else:
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	777 return clrev(node)
229d23cdb203 exchangev2: support fetching shallow files history Gregory Szorc <gregory.szorc@gmail.com> parents: 40330 diff changeset	778
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	779 store.addgroup(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	780 iterrevisions(objs, header[b'totalitems'], progress),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	781 getlinkrev,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	782 weakref.proxy(tr),
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	783 maybemissingparents=shallow,
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	784 )

Mercurial > public > mercurial-scm > hg

annotate mercurial/exchangev2.py @ 45788:a5206e71c536