Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/changegroup.py @ 39883:3e896b51aa5d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
storageutil: move metadata parsing and packing from revlog (API) Parsing and writing of revision text metadata is likely identical across storage backends. Let's move the code out of revlog so we don't need to import the revlog module in order to use it. Differential Revision: https://phab.mercurial-scm.org/D4754
author Gregory Szorc <gregory.szorc@gmail.com>
date Mon, 24 Sep 2018 14:31:31 -0700
parents e23c03dc5cf9
children 8e398628a3f2
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
1 # changegroup.py - Mercurial changegroup manipulation functions
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
2 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
3 # Copyright 2006 Matt Mackall <mpm@selenic.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
4 #
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
5 # This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9437
diff changeset
6 # GNU General Public License version 2 or any later version.
3877
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3859
diff changeset
7
25921
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
8 from __future__ import absolute_import
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
9
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
10 import os
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
11 import struct
20933
d3775db748a0 localrepo: move the addchangegroup method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20932
diff changeset
12 import weakref
25921
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
13
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
14 from .i18n import _
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
15 from .node import (
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
16 hex,
38922
ee1ea96cf9c9 changegroup: move ellipsisdata() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38874
diff changeset
17 nullid,
25921
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
18 nullrev,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
19 short,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
20 )
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
21
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
22 from . import (
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
23 error,
38834
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
24 match as matchmod,
25921
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
25 mdiff,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
26 phases,
30945
82f1ef8b4477 py3: convert the mode argument of os.fdopen to unicodes (2 of 2)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30743
diff changeset
27 pycompat,
38874
a232e6744ba3 narrow: move requirement constant from changegroup to repository
Martin von Zweigbergk <martinvonz@google.com>
parents: 38846
diff changeset
28 repository,
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
29 revlog,
25921
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
30 util,
74b303a637bc changegroup: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25831
diff changeset
31 )
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
32
38935
271854adc3a6 changegroup: make delta header struct formatters actual structs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38934
diff changeset
33 _CHANGEGROUPV1_DELTA_HEADER = struct.Struct("20s20s20s20s")
271854adc3a6 changegroup: make delta header struct formatters actual structs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38934
diff changeset
34 _CHANGEGROUPV2_DELTA_HEADER = struct.Struct("20s20s20s20s20s")
271854adc3a6 changegroup: make delta header struct formatters actual structs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38934
diff changeset
35 _CHANGEGROUPV3_DELTA_HEADER = struct.Struct(">20s20s20s20s20sH")
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
36
37135
a54113fcc8c9 lfs: move the 'supportedoutgoingversions' handling to changegroup.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 37087
diff changeset
37 LFS_REQUIREMENT = 'lfs'
a54113fcc8c9 lfs: move the 'supportedoutgoingversions' handling to changegroup.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 37087
diff changeset
38
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35051
diff changeset
39 readexactly = util.readexactly
13457
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
40
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
41 def getchunk(stream):
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
42 """return the next chunk from stream as a string"""
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
43 d = readexactly(stream, 4)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
44 l = struct.unpack(">l", d)[0]
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
45 if l <= 4:
13458
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
46 if l:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
47 raise error.Abort(_("invalid chunk length %d") % l)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
48 return ""
13457
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
49 return readexactly(stream, l - 4)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
50
5368
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
51 def chunkheader(length):
9437
1c4e4004f3a6 Improve some docstrings relating to changegroups and prepush().
Greg Ward <greg-hg@gerg.ca>
parents: 9087
diff changeset
52 """return a changegroup chunk header (string)"""
5368
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
53 return struct.pack(">l", length + 4)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
54
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
55 def closechunk():
9437
1c4e4004f3a6 Improve some docstrings relating to changegroups and prepush().
Greg Ward <greg-hg@gerg.ca>
parents: 9087
diff changeset
56 """return a changegroup chunk header (string) for a zero-length chunk"""
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
57 return struct.pack(">l", 0)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
58
39019
227ebd88ce5e changegroup: pull _fileheader out of cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39018
diff changeset
59 def _fileheader(path):
227ebd88ce5e changegroup: pull _fileheader out of cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39018
diff changeset
60 """Obtain a changegroup chunk header for a named path."""
227ebd88ce5e changegroup: pull _fileheader out of cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39018
diff changeset
61 return chunkheader(len(path)) + path
227ebd88ce5e changegroup: pull _fileheader out of cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39018
diff changeset
62
26540
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
63 def writechunks(ui, chunks, filename, vfs=None):
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
64 """Write chunks to a file and return its filename.
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
65
26540
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
66 The stream is assumed to be a bundle file.
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
67 Existing files will not be overwritten.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
68 If no filename is specified, a temporary file is created.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
69 """
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
70 fh = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
71 cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
72 try:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
73 if filename:
20976
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
74 if vfs:
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
75 fh = vfs.open(filename, "wb")
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
76 else:
30212
260af19891f2 changegroup: increase write buffer size to 128k
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30211
diff changeset
77 # Increase default buffer size because default is usually
260af19891f2 changegroup: increase write buffer size to 128k
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30211
diff changeset
78 # small (4k is common on Linux).
260af19891f2 changegroup: increase write buffer size to 128k
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30211
diff changeset
79 fh = open(filename, "wb", 131072)
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
80 else:
38197
aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path
Yuya Nishihara <yuya@tcha.org>
parents: 37339
diff changeset
81 fd, filename = pycompat.mkstemp(prefix="hg-bundle-", suffix=".hg")
36843
5bc7ff103081 py3: use r'' instead of sysstr('') to get around code transformer
Yuya Nishihara <yuya@tcha.org>
parents: 36768
diff changeset
82 fh = os.fdopen(fd, r"wb")
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
83 cleanup = filename
26540
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
84 for c in chunks:
7469067de2ba changegroup: extract the file management part in its own function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26509
diff changeset
85 fh.write(c)
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
86 cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
87 return filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
88 finally:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
89 if fh is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
90 fh.close()
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
91 if cleanup is not None:
20976
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
92 if filename and vfs:
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
93 vfs.unlink(cleanup)
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
94 else:
c20f4898631e changegroup: add "vfs" argument to "writebundle()" for relative access via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20966
diff changeset
95 os.unlink(cleanup)
3660
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
96
22390
e2806b8613ca changegroup: rename bundle-related functions and classes
Sune Foldager <cryo@cyanite.org>
parents: 22070
diff changeset
97 class cg1unpacker(object):
26708
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
98 """Unpacker for cg1 changegroup streams.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
99
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
100 A changegroup unpacker handles the framing of the revision data in
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
101 the wire format. Most consumers will want to use the apply()
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
102 method to add the changes from the changegroup to a repository.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
103
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
104 If you're forwarding a changegroup unmodified to another consumer,
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
105 use getchunks(), which returns an iterator of changegroup
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
106 chunks. This is mostly useful for cases where you need to know the
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
107 data stream has ended by observing the end of the changegroup.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
108
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
109 deltachunk() is useful only if you're applying delta data. Most
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
110 consumers should prefer apply() instead.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
111
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
112 A few other public methods exist. Those are used only for
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
113 bundlerepo and some debug commands - their use is discouraged.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
114 """
22390
e2806b8613ca changegroup: rename bundle-related functions and classes
Sune Foldager <cryo@cyanite.org>
parents: 22070
diff changeset
115 deltaheader = _CHANGEGROUPV1_DELTA_HEADER
38935
271854adc3a6 changegroup: make delta header struct formatters actual structs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38934
diff changeset
116 deltaheadersize = deltaheader.size
23896
becfecaf9087 changegroup.writebundle: HG2Y support
Eric Sumner <ericsumner@fb.com>
parents: 23895
diff changeset
117 version = '01'
27920
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
118 _grouplistcount = 1 # One list of files after the manifests
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
119
29593
953839de96ab bundle2: store changeset count when creating file bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29371
diff changeset
120 def __init__(self, fh, alg, extras=None):
30364
a37a96d838b9 changegroup: use compression engines API
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30349
diff changeset
121 if alg is None:
a37a96d838b9 changegroup: use compression engines API
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30349
diff changeset
122 alg = 'UN'
a37a96d838b9 changegroup: use compression engines API
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30349
diff changeset
123 if alg not in util.compengines.supportedbundletypes:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
124 raise error.Abort(_('unknown stream compression type: %s')
26266
1e042e31bd0c changegroup: move all compressions utilities in util
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25921
diff changeset
125 % alg)
26392
127b59787fd5 changegroup: use a different compression key for BZ in HG10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26272
diff changeset
126 if alg == 'BZ':
127b59787fd5 changegroup: use a different compression key for BZ in HG10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26272
diff changeset
127 alg = '_truncatedBZ'
30364
a37a96d838b9 changegroup: use compression engines API
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30349
diff changeset
128
a37a96d838b9 changegroup: use compression engines API
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30349
diff changeset
129 compengine = util.compengines.forbundletype(alg)
a37a96d838b9 changegroup: use compression engines API
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30349
diff changeset
130 self._stream = compengine.decompressorreader(fh)
12044
bcc7139521b7 bundlerepo: remove duplication of bundle decompressors
Matt Mackall <mpm@selenic.com>
parents: 12043
diff changeset
131 self._type = alg
29593
953839de96ab bundle2: store changeset count when creating file bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29371
diff changeset
132 self.extras = extras or {}
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
133 self.callback = None
26706
8c0c3059f478 changegroup: note why a few methods on cg1unpacker exist
Augie Fackler <augie@google.com>
parents: 26704
diff changeset
134
8c0c3059f478 changegroup: note why a few methods on cg1unpacker exist
Augie Fackler <augie@google.com>
parents: 26704
diff changeset
135 # These methods (compressed, read, seek, tell) all appear to only
8c0c3059f478 changegroup: note why a few methods on cg1unpacker exist
Augie Fackler <augie@google.com>
parents: 26704
diff changeset
136 # be used by bundlerepo, but it's a little hard to tell.
12044
bcc7139521b7 bundlerepo: remove duplication of bundle decompressors
Matt Mackall <mpm@selenic.com>
parents: 12043
diff changeset
137 def compressed(self):
30594
182cacaa4c32 cg1packer: fix `compressed` method
Stanislau Hlebik <stash@fb.com>
parents: 30364
diff changeset
138 return self._type is not None and self._type != 'UN'
12043
bef5effb3db0 bundle: introduce bundle class
Matt Mackall <mpm@selenic.com>
parents: 12042
diff changeset
139 def read(self, l):
bef5effb3db0 bundle: introduce bundle class
Matt Mackall <mpm@selenic.com>
parents: 12042
diff changeset
140 return self._stream.read(l)
12330
e527b8635881 bundle: make unbundle object seekable
Matt Mackall <mpm@selenic.com>
parents: 12329
diff changeset
141 def seek(self, pos):
e527b8635881 bundle: make unbundle object seekable
Matt Mackall <mpm@selenic.com>
parents: 12329
diff changeset
142 return self._stream.seek(pos)
e527b8635881 bundle: make unbundle object seekable
Matt Mackall <mpm@selenic.com>
parents: 12329
diff changeset
143 def tell(self):
12332
680fe77ab5b8 bundlerepo: use bundle objects everywhere
Matt Mackall <mpm@selenic.com>
parents: 12330
diff changeset
144 return self._stream.tell()
12347
6277a9469dff bundlerepo: restore close() method
Matt Mackall <mpm@selenic.com>
parents: 12336
diff changeset
145 def close(self):
6277a9469dff bundlerepo: restore close() method
Matt Mackall <mpm@selenic.com>
parents: 12336
diff changeset
146 return self._stream.close()
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
147
26707
5ee6bd529300 changegroup: mark cg1unpacker.chunklength as private
Augie Fackler <augie@google.com>
parents: 26706
diff changeset
148 def _chunklength(self):
13459
acbe171c8fbe changegroup: fix typo introduced in 9f2c407caf34
Jim Hague <jim.hague@acm.org>
parents: 13458
diff changeset
149 d = readexactly(self._stream, 4)
13458
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
150 l = struct.unpack(">l", d)[0]
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
151 if l <= 4:
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
152 if l:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
153 raise error.Abort(_("invalid chunk length %d") % l)
13458
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
154 return 0
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
155 if self.callback:
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
156 self.callback()
13458
9f2c407caf34 changegroup: don't accept odd chunk headers
Mads Kiilerich <mads@kiilerich.com>
parents: 13457
diff changeset
157 return l - 4
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
158
14144
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
159 def changelogheader(self):
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
160 """v10 does not have a changelog header chunk"""
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
161 return {}
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
162
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
163 def manifestheader(self):
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
164 """v10 does not have a manifest header chunk"""
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
165 return {}
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
166
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
167 def filelogheader(self):
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
168 """return the header of the filelogs chunk, v10 only has the filename"""
26707
5ee6bd529300 changegroup: mark cg1unpacker.chunklength as private
Augie Fackler <augie@google.com>
parents: 26706
diff changeset
169 l = self._chunklength()
14144
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
170 if not l:
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
171 return {}
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
172 fname = readexactly(self._stream, l)
20675
f8d50add83e1 changegroup: move from dict() construction to {} literals
Augie Fackler <raf@durin42.com>
parents: 19708
diff changeset
173 return {'filename': fname}
12334
50946802593d bundle: refactor progress callback
Matt Mackall <mpm@selenic.com>
parents: 12333
diff changeset
174
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
175 def _deltaheader(self, headertuple, prevnode):
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
176 node, p1, p2, cs = headertuple
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
177 if prevnode is None:
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
178 deltabase = p1
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
179 else:
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
180 deltabase = prevnode
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
181 flags = 0
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
182 return node, p1, p2, deltabase, cs, flags
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
183
14144
3c3c53d8343a unbundler: separate delta and header parsing
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14143
diff changeset
184 def deltachunk(self, prevnode):
26707
5ee6bd529300 changegroup: mark cg1unpacker.chunklength as private
Augie Fackler <augie@google.com>
parents: 26706
diff changeset
185 l = self._chunklength()
12336
9d234f7d8a77 bundle: move chunk parsing into unbundle class
Matt Mackall <mpm@selenic.com>
parents: 12335
diff changeset
186 if not l:
9d234f7d8a77 bundle: move chunk parsing into unbundle class
Matt Mackall <mpm@selenic.com>
parents: 12335
diff changeset
187 return {}
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
188 headerdata = readexactly(self._stream, self.deltaheadersize)
38935
271854adc3a6 changegroup: make delta header struct formatters actual structs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38934
diff changeset
189 header = self.deltaheader.unpack(headerdata)
14141
bd1cbfe5db5c bundler: make parsechunk return the base revision of the delta
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 14060
diff changeset
190 delta = readexactly(self._stream, l - self.deltaheadersize)
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
191 node, p1, p2, deltabase, cs, flags = self._deltaheader(header, prevnode)
34301
05131c963767 changegroup: remove dictionary creation from deltachunk
Durham Goode <durham@fb.com>
parents: 34298
diff changeset
192 return (node, p1, p2, cs, deltabase, delta, flags)
12336
9d234f7d8a77 bundle: move chunk parsing into unbundle class
Matt Mackall <mpm@selenic.com>
parents: 12335
diff changeset
193
20999
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
194 def getchunks(self):
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
195 """returns all the chunks contains in the bundle
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
196
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
197 Used when you need to forward the binary stream to a file or another
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
198 network API. To do so, it parse the changegroup data, otherwise it will
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
199 block in case of sshrepo because it don't know the end of the stream.
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
200 """
34106
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
201 # For changegroup 1 and 2, we expect 3 parts: changelog, manifestlog,
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
202 # and a list of filelogs. For changegroup 3, we expect 4 parts:
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
203 # changelog, manifestlog, a list of tree manifestlogs, and a list of
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
204 # filelogs.
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
205 #
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
206 # Changelog and manifestlog parts are terminated with empty chunks. The
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
207 # tree and file parts are a list of entry sections. Each entry section
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
208 # is a series of chunks terminating in an empty chunk. The list of these
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
209 # entry sections is terminated in yet another empty chunk, so we know
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
210 # we've reached the end of the tree/file list when we reach an empty
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
211 # chunk that was proceeded by no non-empty chunks.
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
212
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
213 parts = 0
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
214 while parts < 2 + self._grouplistcount:
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
215 noentries = True
20999
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
216 while True:
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
217 chunk = getchunk(self)
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
218 if not chunk:
34106
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
219 # The first two empty chunks represent the end of the
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
220 # changelog and the manifestlog portions. The remaining
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
221 # empty chunks represent either A) the end of individual
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
222 # tree or file entries in the file list, or B) the end of
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
223 # the entire list. It's the end of the entire list if there
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
224 # were no entries (i.e. noentries is True).
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
225 if parts < 2:
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
226 parts += 1
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
227 elif noentries:
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
228 parts += 1
20999
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
229 break
34106
bbdca7e460c0 changegroup: fix to allow empty manifest parts
Durham Goode <durham@fb.com>
parents: 33738
diff changeset
230 noentries = False
20999
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
231 yield chunkheader(len(chunk))
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
232 pos = 0
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
233 while pos < len(chunk):
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
234 next = pos + 2**20
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
235 yield chunk[pos:next]
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
236 pos = next
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
237 yield closechunk()
1e28ec9744bf changegroup: move chunk extraction into a getchunks method of unbundle10
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20978
diff changeset
238
38352
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
239 def _unpackmanifests(self, repo, revmap, trp, prog):
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
240 self.callback = prog.increment
26712
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
241 # no need to check for empty manifest group here:
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
242 # if the result of the merge of 1 and 2 is the same in 3 and 4,
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
243 # no new manifest will be created and the manifest group will
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
244 # be empty during the pull
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
245 self.manifestheader()
34298
1db9abf407c5 revlog: add revmap back to revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34264
diff changeset
246 deltas = self.deltaiter()
39272
73cf21b2e8a6 manifest: add getstorage() to manifestlog and use it globally
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39271
diff changeset
247 repo.manifestlog.getstorage(b'').addgroup(deltas, revmap, trp)
38379
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38352
diff changeset
248 prog.complete()
28360
11287888ce4b changegroup: exclude submanifests from manifest progress
Martin von Zweigbergk <martinvonz@google.com>
parents: 28281
diff changeset
249 self.callback = None
26712
04176eaf911b changegroup: move manifest unpacking into its own method
Augie Fackler <augie@google.com>
parents: 26711
diff changeset
250
33308
248d5890c80a changegroup: remove option to allow empty changegroup (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 33227
diff changeset
251 def apply(self, repo, tr, srctype, url, targetphase=phases.draft,
248d5890c80a changegroup: remove option to allow empty changegroup (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 33227
diff changeset
252 expectedtotal=None):
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
253 """Add the changegroup returned by source.read() to this repo.
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
254 srctype is a string like 'push', 'pull', or 'unbundle'. url is
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
255 the URL of the repo where this changegroup is coming from.
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
256
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
257 Return an integer summarizing the change to this repo:
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
258 - nothing changed or no source: 0
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
259 - more heads than before: 1+added heads (2..n)
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
260 - fewer heads than before: -1-removed heads (-2..-n)
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
261 - number of heads stays the same: 1
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
262 """
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
263 repo = repo.unfiltered()
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
264 def csmap(x):
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
265 repo.ui.debug("add changeset %s\n" % short(x))
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
266 return len(cl)
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
267
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
268 def revmap(x):
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
269 return cl.rev(x)
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
270
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
271 changesets = files = revisions = 0
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
272
26880
fa7f8b686633 changegroup: fix the scope of a try finally
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26859
diff changeset
273 try:
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
274 # The transaction may already carry source information. In this
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
275 # case we use the top level data. We overwrite the argument
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
276 # because we need to use the top level value (if they exist)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
277 # in this function.
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
278 srctype = tr.hookargs.setdefault('source', srctype)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
279 url = tr.hookargs.setdefault('url', url)
33697
38fc45721334 changegroup: wrap some ** expansions in strkwargs
Augie Fackler <augie@google.com>
parents: 33461
diff changeset
280 repo.hook('prechangegroup',
38fc45721334 changegroup: wrap some ** expansions in strkwargs
Augie Fackler <augie@google.com>
parents: 33461
diff changeset
281 throw=True, **pycompat.strkwargs(tr.hookargs))
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
282
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
283 # write changelog data to temp files so concurrent readers
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
284 # will not see an inconsistent view
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
285 cl = repo.changelog
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
286 cl.delayupdate(tr)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
287 oldheads = set(cl.heads())
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
288
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
289 trp = weakref.proxy(tr)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
290 # pull off the changeset group
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
291 repo.ui.status(_("adding changesets\n"))
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
292 clstart = len(cl)
38352
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
293 progress = repo.ui.makeprogress(_('changesets'), unit=_('chunks'),
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
294 total=expectedtotal)
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
295 self.callback = progress.increment
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
296
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
297 efiles = set()
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
298 def onchangelog(cl, node):
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
299 efiles.update(cl.readfiles(node))
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
300
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
301 self.changelogheader()
34298
1db9abf407c5 revlog: add revmap back to revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34264
diff changeset
302 deltas = self.deltaiter()
1db9abf407c5 revlog: add revmap back to revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34264
diff changeset
303 cgnodes = cl.addgroup(deltas, csmap, trp, addrevisioncb=onchangelog)
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
304 efiles = len(efiles)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
305
33308
248d5890c80a changegroup: remove option to allow empty changegroup (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 33227
diff changeset
306 if not cgnodes:
39686
9b07ee0a1054 changegroup: improve the devel-warn to specify changelog was empty
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39342
diff changeset
307 repo.ui.develwarn('applied empty changelog from changegroup',
34734
3572b2031cec devel-warn: add 'warn-' to 'devel.empty-changegroup' config
Boris Feld <boris.feld@octobus.net>
parents: 34301
diff changeset
308 config='warn-empty-changegroup')
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
309 clend = len(cl)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
310 changesets = clend - clstart
38379
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38352
diff changeset
311 progress.complete()
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
312 self.callback = None
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
313
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
314 # pull off the manifest group
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
315 repo.ui.status(_("adding manifests\n"))
38352
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
316 # We know that we'll never have more manifests than we had
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
317 # changesets.
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
318 progress = repo.ui.makeprogress(_('manifests'), unit=_('chunks'),
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
319 total=changesets)
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
320 self._unpackmanifests(repo, revmap, trp, progress)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
321
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
322 needfiles = {}
33227
86c9aa1d598f configitems: register the 'server.validate' config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33185
diff changeset
323 if repo.ui.configbool('server', 'validate'):
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
324 cl = repo.changelog
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
325 ml = repo.manifestlog
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
326 # validate incoming csets have their manifests
38823
e7aa113b14f7 global: use pycompat.xrange()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38557
diff changeset
327 for cset in pycompat.xrange(clstart, clend):
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
328 mfnode = cl.changelogrevision(cset).manifest
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
329 mfest = ml[mfnode].readdelta()
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
330 # store file cgnodes we must see
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
331 for f, n in mfest.iteritems():
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
332 needfiles.setdefault(f, set()).add(n)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
333
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
334 # process the files
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
335 repo.ui.status(_("adding file changes\n"))
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
336 newrevs, newfiles = _addchangegroupfiles(
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
337 repo, self, revmap, trp, efiles, needfiles)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
338 revisions += newrevs
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
339 files += newfiles
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
340
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
341 deltaheads = 0
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
342 if oldheads:
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
343 heads = cl.heads()
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
344 deltaheads = len(heads) - len(oldheads)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
345 for h in heads:
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
346 if h not in oldheads and repo[h].closesbranch():
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
347 deltaheads -= 1
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
348 htext = ""
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
349 if deltaheads:
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
350 htext = _(" (%+d heads)") % deltaheads
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
351
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
352 repo.ui.status(_("added %d changesets"
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
353 " with %d changes to %d files%s\n")
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
354 % (changesets, revisions, files, htext))
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
355 repo.invalidatevolatilesets()
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
356
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
357 if changesets > 0:
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
358 if 'node' not in tr.hookargs:
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
359 tr.hookargs['node'] = hex(cl.node(clstart))
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
360 tr.hookargs['node_last'] = hex(cl.node(clend - 1))
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
361 hookargs = dict(tr.hookargs)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
362 else:
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
363 hookargs = dict(tr.hookargs)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
364 hookargs['node'] = hex(cl.node(clstart))
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
365 hookargs['node_last'] = hex(cl.node(clend - 1))
33697
38fc45721334 changegroup: wrap some ** expansions in strkwargs
Augie Fackler <augie@google.com>
parents: 33461
diff changeset
366 repo.hook('pretxnchangegroup',
38fc45721334 changegroup: wrap some ** expansions in strkwargs
Augie Fackler <augie@google.com>
parents: 33461
diff changeset
367 throw=True, **pycompat.strkwargs(hookargs))
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
368
38823
e7aa113b14f7 global: use pycompat.xrange()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38557
diff changeset
369 added = [cl.node(r) for r in pycompat.xrange(clstart, clend)]
33456
ae052d04b89e phases: rework phase movement code in 'cg.apply' to use 'registernew'
Boris Feld <boris.feld@octobus.net>
parents: 33406
diff changeset
370 phaseall = None
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
371 if srctype in ('push', 'serve'):
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
372 # Old servers can not push the boundary themselves.
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
373 # New servers won't push the boundary if changeset already
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
374 # exists locally as secret
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
375 #
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
376 # We should not use added here but the list of all change in
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
377 # the bundle
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
378 if repo.publishing():
33456
ae052d04b89e phases: rework phase movement code in 'cg.apply' to use 'registernew'
Boris Feld <boris.feld@octobus.net>
parents: 33406
diff changeset
379 targetphase = phaseall = phases.public
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
380 else:
33456
ae052d04b89e phases: rework phase movement code in 'cg.apply' to use 'registernew'
Boris Feld <boris.feld@octobus.net>
parents: 33406
diff changeset
381 # closer target phase computation
ae052d04b89e phases: rework phase movement code in 'cg.apply' to use 'registernew'
Boris Feld <boris.feld@octobus.net>
parents: 33406
diff changeset
382
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
383 # Those changesets have been pushed from the
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
384 # outside, their phases are going to be pushed
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
385 # alongside. Therefor `targetphase` is
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
386 # ignored.
33456
ae052d04b89e phases: rework phase movement code in 'cg.apply' to use 'registernew'
Boris Feld <boris.feld@octobus.net>
parents: 33406
diff changeset
387 targetphase = phaseall = phases.draft
ae052d04b89e phases: rework phase movement code in 'cg.apply' to use 'registernew'
Boris Feld <boris.feld@octobus.net>
parents: 33406
diff changeset
388 if added:
ae052d04b89e phases: rework phase movement code in 'cg.apply' to use 'registernew'
Boris Feld <boris.feld@octobus.net>
parents: 33406
diff changeset
389 phases.registernew(repo, tr, targetphase, added)
ae052d04b89e phases: rework phase movement code in 'cg.apply' to use 'registernew'
Boris Feld <boris.feld@octobus.net>
parents: 33406
diff changeset
390 if phaseall is not None:
ae052d04b89e phases: rework phase movement code in 'cg.apply' to use 'registernew'
Boris Feld <boris.feld@octobus.net>
parents: 33406
diff changeset
391 phases.advanceboundary(repo, tr, phaseall, cgnodes)
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
392
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
393 if changesets > 0:
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
394
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
395 def runhooks():
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
396 # These hooks run when the lock releases, not when the
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
397 # transaction closes. So it's possible for the changelog
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
398 # to have changed since we last saw it.
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
399 if clstart >= len(repo):
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
400 return
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
401
33738
672ad4f3bb84 changegroup: more **kwargs
Augie Fackler <augie@google.com>
parents: 33697
diff changeset
402 repo.hook("changegroup", **pycompat.strkwargs(hookargs))
27867
7ced54ebf972 with: use context manager for transaction in changegroup apply
Bryan O'Sullivan <bryano@fb.com>
parents: 27754
diff changeset
403
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
404 for n in added:
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
405 args = hookargs.copy()
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
406 args['node'] = hex(n)
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
407 del args['node_last']
33738
672ad4f3bb84 changegroup: more **kwargs
Augie Fackler <augie@google.com>
parents: 33697
diff changeset
408 repo.hook("incoming", **pycompat.strkwargs(args))
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
409
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
410 newheads = [h for h in repo.heads()
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
411 if h not in oldheads]
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
412 repo.ui.log("incoming",
36760
2a5024109490 py3: fix int formatting of "incoming changes" log
Yuya Nishihara <yuya@tcha.org>
parents: 36495
diff changeset
413 "%d incoming changes - new heads: %s\n",
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
414 len(added),
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
415 ', '.join([hex(c[:6]) for c in newheads]))
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
416
32949
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
417 tr.addpostclose('changegroup-runhooks-%020i' % clstart,
b08431e1b062 changegroup: delete "if True" and reflow
Martin von Zweigbergk <martinvonz@google.com>
parents: 32948
diff changeset
418 lambda tr: repo._afterlock(runhooks))
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
419 finally:
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
420 repo.ui.flush()
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
421 # never return 0 here:
32888
b441296f8e9c changegroup: rename "dh" to the clearer "deltaheads"
Martin von Zweigbergk <martinvonz@google.com>
parents: 32887
diff changeset
422 if deltaheads < 0:
33042
3e102a8dd52c bundle2: record changegroup data in 'op.records' (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 32949
diff changeset
423 ret = deltaheads - 1
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
424 else:
33042
3e102a8dd52c bundle2: record changegroup data in 'op.records' (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 32949
diff changeset
425 ret = deltaheads + 1
33461
bb72031f0ea8 changegroup: stop returning and recording added nodes in 'cg.apply'
Boris Feld <boris.feld@octobus.net>
parents: 33456
diff changeset
426 return ret
26695
1121fced5b20 changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()
Augie Fackler <augie@google.com>
parents: 26694
diff changeset
427
34298
1db9abf407c5 revlog: add revmap back to revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34264
diff changeset
428 def deltaiter(self):
34160
c8b6ed51386b changegroup: remove changegroup dependency from revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34118
diff changeset
429 """
c8b6ed51386b changegroup: remove changegroup dependency from revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34118
diff changeset
430 returns an iterator of the deltas in this changegroup
c8b6ed51386b changegroup: remove changegroup dependency from revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34118
diff changeset
431
c8b6ed51386b changegroup: remove changegroup dependency from revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34118
diff changeset
432 Useful for passing to the underlying storage system to be stored.
c8b6ed51386b changegroup: remove changegroup dependency from revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34118
diff changeset
433 """
c8b6ed51386b changegroup: remove changegroup dependency from revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34118
diff changeset
434 chain = None
c8b6ed51386b changegroup: remove changegroup dependency from revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34118
diff changeset
435 for chunkdata in iter(lambda: self.deltachunk(chain), {}):
34301
05131c963767 changegroup: remove dictionary creation from deltachunk
Durham Goode <durham@fb.com>
parents: 34298
diff changeset
436 # Chunkdata: (node, p1, p2, cs, deltabase, delta, flags)
05131c963767 changegroup: remove dictionary creation from deltachunk
Durham Goode <durham@fb.com>
parents: 34298
diff changeset
437 yield chunkdata
05131c963767 changegroup: remove dictionary creation from deltachunk
Durham Goode <durham@fb.com>
parents: 34298
diff changeset
438 chain = chunkdata[0]
34160
c8b6ed51386b changegroup: remove changegroup dependency from revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34118
diff changeset
439
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
440 class cg2unpacker(cg1unpacker):
26708
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
441 """Unpacker for cg2 streams.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
442
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
443 cg2 streams add support for generaldelta, so the delta header
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
444 format is slightly different. All other features about the data
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
445 remain the same.
749d913f24b8 changegroup: document the public surface area of cg?unpackers
Augie Fackler <augie@google.com>
parents: 26707
diff changeset
446 """
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
447 deltaheader = _CHANGEGROUPV2_DELTA_HEADER
38935
271854adc3a6 changegroup: make delta header struct formatters actual structs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38934
diff changeset
448 deltaheadersize = deltaheader.size
23896
becfecaf9087 changegroup.writebundle: HG2Y support
Eric Sumner <ericsumner@fb.com>
parents: 23895
diff changeset
449 version = '02'
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
450
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
451 def _deltaheader(self, headertuple, prevnode):
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
452 node, p1, p2, deltabase, cs = headertuple
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
453 flags = 0
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
454 return node, p1, p2, deltabase, cs, flags
23181
832b7ef275c8 changegroup: introduce cg2packer/unpacker
Sune Foldager <cryo@cyanite.org>
parents: 23178
diff changeset
455
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
456 class cg3unpacker(cg2unpacker):
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
457 """Unpacker for cg3 streams.
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
458
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
459 cg3 streams add support for exchanging treemanifests and revlog
27753
d4071cc73f46 changegroup3: add empty chunk separating directories and files
Martin von Zweigbergk <martinvonz@google.com>
parents: 27752
diff changeset
460 flags. It adds the revlog flags to the delta header and an empty chunk
d4071cc73f46 changegroup3: add empty chunk separating directories and files
Martin von Zweigbergk <martinvonz@google.com>
parents: 27752
diff changeset
461 separating manifests and files.
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
462 """
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
463 deltaheader = _CHANGEGROUPV3_DELTA_HEADER
38935
271854adc3a6 changegroup: make delta header struct formatters actual structs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38934
diff changeset
464 deltaheadersize = deltaheader.size
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
465 version = '03'
27920
da5f23362517 changegroup: cg3 has two empty groups *after* manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27867
diff changeset
466 _grouplistcount = 2 # One list of manifests and one list of files
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
467
27433
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
468 def _deltaheader(self, headertuple, prevnode):
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
469 node, p1, p2, deltabase, cs, flags = headertuple
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
470 return node, p1, p2, deltabase, cs, flags
12f727a5b434 changegroup: add flags field to cg3 delta header
Mike Edgar <adgar@google.com>
parents: 27432
diff changeset
471
38352
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
472 def _unpackmanifests(self, repo, revmap, trp, prog):
83534c4ec58b changegroup: use progress helper in apply() (API)
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
473 super(cg3unpacker, self)._unpackmanifests(repo, revmap, trp, prog)
29736
4e7be6e33269 changegroup: use `iter(callable, sentinel)` instead of while True
Augie Fackler <augie@google.com>
parents: 29704
diff changeset
474 for chunkdata in iter(self.filelogheader, {}):
27754
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
475 # If we get here, there are directory manifests in the changegroup
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
476 d = chunkdata["filename"]
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
477 repo.ui.debug("adding %s revisions\n" % d)
34298
1db9abf407c5 revlog: add revmap back to revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34264
diff changeset
478 deltas = self.deltaiter()
39272
73cf21b2e8a6 manifest: add getstorage() to manifestlog and use it globally
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39271
diff changeset
479 if not repo.manifestlog.getstorage(d).addgroup(deltas, revmap, trp):
27754
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
480 raise error.Abort(_("received dir revlog group is empty"))
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
481
12329
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
482 class headerlessfixup(object):
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
483 def __init__(self, fh, h):
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
484 self._h = h
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
485 self._fh = fh
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
486 def read(self, n):
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
487 if self._h:
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
488 d, self._h = self._h[:n], self._h[n:]
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
489 if len(d) < n:
13457
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
490 d += readexactly(self._fh, n - len(d))
12329
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
491 return d
13457
e74fe15dc7fd changegroup: verify all stream reads
Mads Kiilerich <mads@kiilerich.com>
parents: 13456
diff changeset
492 return readexactly(self._fh, n)
12329
7458de933f26 bundle: push chunkbuffer down into decompress
Matt Mackall <mpm@selenic.com>
parents: 12044
diff changeset
493
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
494 def _revisiondeltatochunks(delta, headerfn):
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
495 """Serialize a revisiondelta to changegroup chunks."""
39054
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
496
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
497 # The captured revision delta may be encoded as a delta against
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
498 # a base revision or as a full revision. The changegroup format
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
499 # requires that everything on the wire be deltas. So for full
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
500 # revisions, we need to invent a header that says to rewrite
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
501 # data.
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
502
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
503 if delta.delta is not None:
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
504 prefix, data = b'', delta.delta
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
505 elif delta.basenode == nullid:
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
506 data = delta.revision
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
507 prefix = mdiff.trivialdiffheader(len(data))
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
508 else:
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
509 data = delta.revision
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
510 prefix = mdiff.replacediffheader(delta.baserevisionsize,
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
511 len(data))
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
512
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
513 meta = headerfn(delta)
39054
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
514
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
515 yield chunkheader(len(meta) + len(prefix) + len(data))
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
516 yield meta
39054
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
517 if prefix:
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
518 yield prefix
39b8277e2115 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39053
diff changeset
519 yield data
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
520
39035
812eec3f89cb changegroup: remove _clnodetorev
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39034
diff changeset
521 def _sortnodesellipsis(store, nodes, cl, lookup):
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
522 """Sort nodes for changegroup generation."""
39020
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
523 # Ellipses serving mode.
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
524 #
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
525 # In a perfect world, we'd generate better ellipsis-ified graphs
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
526 # for non-changelog revlogs. In practice, we haven't started doing
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
527 # that yet, so the resulting DAGs for the manifestlog and filelogs
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
528 # are actually full of bogus parentage on all the ellipsis
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
529 # nodes. This has the side effect that, while the contents are
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
530 # correct, the individual DAGs might be completely out of whack in
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
531 # a case like 882681bc3166 and its ancestors (back about 10
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
532 # revisions or so) in the main hg repo.
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
533 #
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
534 # The one invariant we *know* holds is that the new (potentially
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
535 # bogus) DAG shape will be valid if we order the nodes in the
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
536 # order that they're introduced in dramatis personae by the
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
537 # changelog, so what we do is we sort the non-changelog histories
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
538 # by the order in which they are used by the changelog.
39035
812eec3f89cb changegroup: remove _clnodetorev
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39034
diff changeset
539 key = lambda n: cl.rev(lookup(n))
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
540 return sorted(nodes, key=key)
39020
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
541
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
542 def _resolvenarrowrevisioninfo(cl, store, ischangelog, rev, linkrev,
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
543 linknode, clrevtolocalrev, fullclnodes,
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
544 precomputedellipsis):
39042
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
545 linkparents = precomputedellipsis[linkrev]
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
546 def local(clrev):
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
547 """Turn a changelog revnum into a local revnum.
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
548
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
549 The ellipsis dag is stored as revnums on the changelog,
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
550 but when we're producing ellipsis entries for
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
551 non-changelog revlogs, we need to turn those numbers into
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
552 something local. This does that for us, and during the
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
553 changelog sending phase will also expand the stored
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
554 mappings as needed.
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
555 """
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
556 if clrev == nullrev:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
557 return nullrev
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
558
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
559 if ischangelog:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
560 return clrev
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
561
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
562 # Walk the ellipsis-ized changelog breadth-first looking for a
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
563 # change that has been linked from the current revlog.
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
564 #
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
565 # For a flat manifest revlog only a single step should be necessary
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
566 # as all relevant changelog entries are relevant to the flat
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
567 # manifest.
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
568 #
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
569 # For a filelog or tree manifest dirlog however not every changelog
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
570 # entry will have been relevant, so we need to skip some changelog
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
571 # nodes even after ellipsis-izing.
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
572 walk = [clrev]
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
573 while walk:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
574 p = walk[0]
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
575 walk = walk[1:]
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
576 if p in clrevtolocalrev:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
577 return clrevtolocalrev[p]
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
578 elif p in fullclnodes:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
579 walk.extend([pp for pp in cl.parentrevs(p)
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
580 if pp != nullrev])
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
581 elif p in precomputedellipsis:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
582 walk.extend([pp for pp in precomputedellipsis[p]
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
583 if pp != nullrev])
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
584 else:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
585 # In this case, we've got an ellipsis with parents
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
586 # outside the current bundle (likely an
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
587 # incremental pull). We "know" that we can use the
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
588 # value of this same revlog at whatever revision
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
589 # is pointed to by linknode. "Know" is in scare
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
590 # quotes because I haven't done enough examination
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
591 # of edge cases to convince myself this is really
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
592 # a fact - it works for all the (admittedly
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
593 # thorough) cases in our testsuite, but I would be
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
594 # somewhat unsurprised to find a case in the wild
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
595 # where this breaks down a bit. That said, I don't
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
596 # know if it would hurt anything.
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
597 for i in pycompat.xrange(rev, 0, -1):
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
598 if store.linkrev(i) == clrev:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
599 return i
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
600 # We failed to resolve a parent for this node, so
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
601 # we crash the changegroup construction.
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
602 raise error.Abort(
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
603 'unable to resolve parent while packing %r %r'
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
604 ' for changeset %r' % (store.indexfile, rev, clrev))
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
605
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
606 return nullrev
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
607
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
608 if not linkparents or (
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
609 store.parentrevs(rev) == (nullrev, nullrev)):
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
610 p1, p2 = nullrev, nullrev
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
611 elif len(linkparents) == 1:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
612 p1, = sorted(local(p) for p in linkparents)
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
613 p2 = nullrev
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
614 else:
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
615 p1, p2 = sorted(local(p) for p in linkparents)
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
616
39056
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
617 p1node, p2node = store.node(p1), store.node(p2)
39042
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
618
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
619 return p1node, p2node, linknode
39042
e11d07cc125c changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39041
diff changeset
620
39257
2646b8d66b7b changegroup: move node sorting into deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39214
diff changeset
621 def deltagroup(repo, store, nodes, ischangelog, lookup, forcedeltaparentprev,
39267
0617a700ef7b changegroup: change topics during generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39266
diff changeset
622 topic=None,
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
623 ellipses=False, clrevtolocalrev=None, fullclnodes=None,
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
624 precomputedellipsis=None):
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
625 """Calculate deltas for a set of revisions.
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
626
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
627 Is a generator of ``revisiondelta`` instances.
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
628
39267
0617a700ef7b changegroup: change topics during generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39266
diff changeset
629 If topic is not None, progress detail will be generated using this
0617a700ef7b changegroup: change topics during generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39266
diff changeset
630 topic name (e.g. changesets, manifests, etc).
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
631 """
39257
2646b8d66b7b changegroup: move node sorting into deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39214
diff changeset
632 if not nodes:
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
633 return
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
634
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
635 cl = repo.changelog
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
636
39257
2646b8d66b7b changegroup: move node sorting into deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39214
diff changeset
637 if ischangelog:
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
638 # `hg log` shows changesets in storage order. To preserve order
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
639 # across clones, send out changesets in storage order.
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
640 nodesorder = 'storage'
39257
2646b8d66b7b changegroup: move node sorting into deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39214
diff changeset
641 elif ellipses:
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
642 nodes = _sortnodesellipsis(store, nodes, cl, lookup)
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
643 nodesorder = 'nodes'
39257
2646b8d66b7b changegroup: move node sorting into deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39214
diff changeset
644 else:
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
645 nodesorder = None
39257
2646b8d66b7b changegroup: move node sorting into deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39214
diff changeset
646
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
647 # Perform ellipses filtering and revision massaging. We do this before
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
648 # emitrevisions() because a) filtering out revisions creates less work
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
649 # for emitrevisions() b) dropping revisions would break emitrevisions()'s
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
650 # assumptions about delta choices and we would possibly send a delta
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
651 # referencing a missing base revision.
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
652 #
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
653 # Also, calling lookup() has side-effects with regards to populating
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
654 # data structures. If we don't call lookup() for each node or if we call
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
655 # lookup() after the first pass through each node, things can break -
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
656 # possibly intermittently depending on the python hash seed! For that
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
657 # reason, we store a mapping of all linknodes during the initial node
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
658 # pass rather than use lookup() on the output side.
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
659 if ellipses:
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
660 filtered = []
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
661 adjustedparents = {}
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
662 linknodes = {}
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
663
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
664 for node in nodes:
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
665 rev = store.rev(node)
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
666 linknode = lookup(node)
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
667 linkrev = cl.rev(linknode)
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
668 clrevtolocalrev[linkrev] = rev
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
669
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
670 # If linknode is in fullclnodes, it means the corresponding
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
671 # changeset was a full changeset and is being sent unaltered.
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
672 if linknode in fullclnodes:
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
673 linknodes[node] = linknode
39056
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
674
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
675 # If the corresponding changeset wasn't in the set computed
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
676 # as relevant to us, it should be dropped outright.
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
677 elif linkrev not in precomputedellipsis:
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
678 continue
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
679
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
680 else:
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
681 # We could probably do this later and avoid the dict
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
682 # holding state. But it likely doesn't matter.
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
683 p1node, p2node, linknode = _resolvenarrowrevisioninfo(
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
684 cl, store, ischangelog, rev, linkrev, linknode,
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
685 clrevtolocalrev, fullclnodes, precomputedellipsis)
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
686
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
687 adjustedparents[node] = (p1node, p2node)
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
688 linknodes[node] = linknode
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
689
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
690 filtered.append(node)
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
691
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
692 nodes = filtered
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
693
39056
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
694 # We expect the first pass to be fast, so we only engage the progress
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
695 # meter for constructing the revision deltas.
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
696 progress = None
39267
0617a700ef7b changegroup: change topics during generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39266
diff changeset
697 if topic is not None:
0617a700ef7b changegroup: change topics during generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39266
diff changeset
698 progress = repo.ui.makeprogress(topic, unit=_('chunks'),
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
699 total=len(nodes))
39056
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
700
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
701 revisions = store.emitrevisions(
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
702 nodes,
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
703 nodesorder=nodesorder,
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
704 revisiondata=True,
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
705 assumehaveparentrevisions=not ellipses,
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
706 deltaprevious=forcedeltaparentprev)
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
707
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
708 for i, revision in enumerate(revisions):
39056
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
709 if progress:
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
710 progress.update(i + 1)
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
711
39870
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
712 if ellipses:
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
713 linknode = linknodes[revision.node]
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
714
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
715 if revision.node in adjustedparents:
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
716 p1node, p2node = adjustedparents[revision.node]
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
717 revision.p1node = p1node
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
718 revision.p2node = p2node
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
719 revision.flags |= revlog.REVIDX_ELLIPSIS
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
720
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
721 else:
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
722 linknode = lookup(revision.node)
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
723
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
724 revision.linknode = linknode
31b7e8e7132e changegroup: port to emitrevisions() (issue5976)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39866
diff changeset
725 yield revision
39056
e793e11e1462 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39055
diff changeset
726
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
727 if progress:
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
728 progress.complete()
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
729
38941
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
730 class cgpacker(object):
39866
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
731 def __init__(self, repo, filematcher, version,
39055
ef3d3a2f9aa5 changegroup: refactor delta parent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39054
diff changeset
732 builddeltaheader, manifestsend,
ef3d3a2f9aa5 changegroup: refactor delta parent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39054
diff changeset
733 forcedeltaparentprev=False,
38964
8a13855c8dbe changegroup: always use the treemanifest-enabled version of _packmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 38949
diff changeset
734 bundlecaps=None, ellipses=False,
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
735 shallow=False, ellipsisroots=None, fullnodes=None):
19202
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
736 """Given a source repo, construct a bundler.
32327
df3cf9422e1b changegroup: add bundlecaps back
Durham Goode <durham@fb.com>
parents: 32308
diff changeset
737
38834
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
738 filematcher is a matcher that matches on files to include in the
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
739 changegroup. Used to facilitate sparse changegroups.
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
740
39055
ef3d3a2f9aa5 changegroup: refactor delta parent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39054
diff changeset
741 forcedeltaparentprev indicates whether delta parents must be against
ef3d3a2f9aa5 changegroup: refactor delta parent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39054
diff changeset
742 the previous revision in a delta group. This should only be used for
ef3d3a2f9aa5 changegroup: refactor delta parent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39054
diff changeset
743 compatibility with changegroup version 1.
38940
23ae0c07a3e1 changegroup: control delta parent behavior via constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38939
diff changeset
744
38936
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
745 builddeltaheader is a callable that constructs the header for a group
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
746 delta.
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
747
38937
67f37e8a5490 changegroup: pass end of manifests marker into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38936
diff changeset
748 manifestsend is a chunk to send after manifests have been fully emitted.
67f37e8a5490 changegroup: pass end of manifests marker into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38936
diff changeset
749
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
750 ellipses indicates whether ellipsis serving mode is enabled.
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
751
32327
df3cf9422e1b changegroup: add bundlecaps back
Durham Goode <durham@fb.com>
parents: 32308
diff changeset
752 bundlecaps is optional and can be used to specify the set of
df3cf9422e1b changegroup: add bundlecaps back
Durham Goode <durham@fb.com>
parents: 32308
diff changeset
753 capabilities which can be used to build the bundle. While bundlecaps is
df3cf9422e1b changegroup: add bundlecaps back
Durham Goode <durham@fb.com>
parents: 32308
diff changeset
754 unused in core Mercurial, extensions rely on this feature to communicate
df3cf9422e1b changegroup: add bundlecaps back
Durham Goode <durham@fb.com>
parents: 32308
diff changeset
755 capabilities to customize the changegroup packer.
38943
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
756
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
757 shallow indicates whether shallow data might be sent. The packer may
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
758 need to pack file contents not introduced by the changes being packed.
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
759
39034
5baafb8fe253 changegroup: rename _fullnodes to _fullclnodes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39033
diff changeset
760 fullnodes is the set of changelog nodes which should not be ellipsis
5baafb8fe253 changegroup: rename _fullnodes to _fullclnodes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39033
diff changeset
761 nodes. We store this rather than the set of nodes that should be
5baafb8fe253 changegroup: rename _fullnodes to _fullclnodes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39033
diff changeset
762 ellipsis because for very large histories we expect this to be
5baafb8fe253 changegroup: rename _fullnodes to _fullclnodes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39033
diff changeset
763 significantly smaller.
19202
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
764 """
38834
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
765 assert filematcher
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
766 self._filematcher = filematcher
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
767
38934
d7ac49c2353c changegroup: pass version into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38933
diff changeset
768 self.version = version
39055
ef3d3a2f9aa5 changegroup: refactor delta parent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39054
diff changeset
769 self._forcedeltaparentprev = forcedeltaparentprev
38936
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
770 self._builddeltaheader = builddeltaheader
38937
67f37e8a5490 changegroup: pass end of manifests marker into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38936
diff changeset
771 self._manifestsend = manifestsend
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
772 self._ellipses = ellipses
38934
d7ac49c2353c changegroup: pass version into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38933
diff changeset
773
32327
df3cf9422e1b changegroup: add bundlecaps back
Durham Goode <durham@fb.com>
parents: 32308
diff changeset
774 # Set of capabilities we can use to build the bundle.
df3cf9422e1b changegroup: add bundlecaps back
Durham Goode <durham@fb.com>
parents: 32308
diff changeset
775 if bundlecaps is None:
df3cf9422e1b changegroup: add bundlecaps back
Durham Goode <durham@fb.com>
parents: 32308
diff changeset
776 bundlecaps = set()
df3cf9422e1b changegroup: add bundlecaps back
Durham Goode <durham@fb.com>
parents: 32308
diff changeset
777 self._bundlecaps = bundlecaps
38943
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
778 self._isshallow = shallow
39034
5baafb8fe253 changegroup: rename _fullnodes to _fullclnodes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39033
diff changeset
779 self._fullclnodes = fullnodes
38939
6e999a2d8fe7 changegroup: control reordering via constructor argument
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38938
diff changeset
780
38946
ad4c4cc9a5ac changegroup: pass ellipsis roots into cgpacker constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38945
diff changeset
781 # Maps ellipsis revs to their roots at the changelog level.
ad4c4cc9a5ac changegroup: pass ellipsis roots into cgpacker constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38945
diff changeset
782 self._precomputedellipsis = ellipsisroots
ad4c4cc9a5ac changegroup: pass ellipsis roots into cgpacker constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38945
diff changeset
783
19202
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
784 self._repo = repo
38939
6e999a2d8fe7 changegroup: control reordering via constructor argument
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38938
diff changeset
785
23748
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
786 if self._repo.ui.verbose and not self._repo.ui.debugflag:
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
787 self._verbosenote = self._repo.ui.note
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
788 else:
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
789 self._verbosenote = lambda s: None
4ab66de46a96 bundle: when verbose, show what takes up the space in the generated bundle
Mads Kiilerich <madski@unity3d.com>
parents: 23382
diff changeset
790
39688
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
791 def generate(self, commonrevs, clnodes, fastpathlinkrev, source,
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
792 changelog=True):
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
793 """Yield a sequence of changegroup byte chunks.
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
794 If changelog is False, changelog data won't be added to changegroup
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
795 """
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
796
19202
0455fc94ae00 bundle-ng: move gengroup into bundler, pass repo object to bundler
Sune Foldager <cryo@cyanite.org>
parents: 19201
diff changeset
797 repo = self._repo
24978
f52560c64953 changegroup: drop _changelog and _manifest properties
Martin von Zweigbergk <martinvonz@google.com>
parents: 24977
diff changeset
798 cl = repo.changelog
19204
e9c5b1c246dc bundle-ng: move bundle generation to changegroup.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19202
diff changeset
799
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
800 self._verbosenote(_('uncompressed size of bundle content:\n'))
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
801 size = 0
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
802
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
803 clstate, deltas = self._generatechangelog(cl, clnodes)
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
804 for delta in deltas:
39688
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
805 if changelog:
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
806 for chunk in _revisiondeltatochunks(delta,
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
807 self._builddeltaheader):
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
808 size += len(chunk)
a1942015c10e changegroup: add functionality to skip adding changelog data to changegroup
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39686
diff changeset
809 yield chunk
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
810
39048
fcdab6629dde changegroup: emit delta group close chunk outside of deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39047
diff changeset
811 close = closechunk()
fcdab6629dde changegroup: emit delta group close chunk outside of deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39047
diff changeset
812 size += len(close)
fcdab6629dde changegroup: emit delta group close chunk outside of deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39047
diff changeset
813 yield closechunk()
fcdab6629dde changegroup: emit delta group close chunk outside of deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39047
diff changeset
814
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
815 self._verbosenote(_('%8.i (changelog)\n') % size)
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
816
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
817 clrevorder = clstate['clrevorder']
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
818 manifests = clstate['manifests']
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
819 changedfiles = clstate['changedfiles']
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
820
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
821 # We need to make sure that the linkrev in the changegroup refers to
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
822 # the first changeset that introduced the manifest or file revision.
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
823 # The fastpath is usually safer than the slowpath, because the filelogs
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
824 # are walked in revlog order.
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
825 #
39866
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
826 # When taking the slowpath when the manifest revlog uses generaldelta,
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
827 # the manifest may be walked in the "wrong" order. Without 'clrevorder',
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
828 # we would get an incorrect linkrev (see fix in cc0ff93d0c0c).
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
829 #
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
830 # When taking the fastpath, we are only vulnerable to reordering
39866
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
831 # of the changelog itself. The changelog never uses generaldelta and is
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
832 # never reordered. To handle this case, we simply take the slowpath,
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
833 # which already has the 'clrevorder' logic. This was also fixed in
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
834 # cc0ff93d0c0c.
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
835
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
836 # Treemanifests don't work correctly with fastpathlinkrev
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
837 # either, because we don't discover which directory nodes to
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
838 # send along with files. This could probably be fixed.
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
839 fastpathlinkrev = fastpathlinkrev and (
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
840 'treemanifest' not in repo.requirements)
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
841
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
842 fnodes = {} # needed file nodes
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
843
39049
2ebdd265fe8c changegroup: move size tracking and end of manifests to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39048
diff changeset
844 size = 0
39050
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
845 it = self.generatemanifests(
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
846 commonrevs, clrevorder, fastpathlinkrev, manifests, fnodes, source,
39050
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
847 clstate['clrevtomanifestrev'])
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
848
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
849 for tree, deltas in it:
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
850 if tree:
39050
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
851 assert self.version == b'03'
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
852 chunk = _fileheader(tree)
39050
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
853 size += len(chunk)
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
854 yield chunk
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
855
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
856 for delta in deltas:
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
857 chunks = _revisiondeltatochunks(delta, self._builddeltaheader)
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
858 for chunk in chunks:
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
859 size += len(chunk)
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
860 yield chunk
39050
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
861
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
862 close = closechunk()
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
863 size += len(close)
c921ad9cae08 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39049
diff changeset
864 yield close
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
865
39049
2ebdd265fe8c changegroup: move size tracking and end of manifests to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39048
diff changeset
866 self._verbosenote(_('%8.i (manifests)\n') % size)
2ebdd265fe8c changegroup: move size tracking and end of manifests to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39048
diff changeset
867 yield self._manifestsend
2ebdd265fe8c changegroup: move size tracking and end of manifests to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39048
diff changeset
868
39021
fbbda9ff3deb changegroup: pass mfdicts properly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39020
diff changeset
869 mfdicts = None
fbbda9ff3deb changegroup: pass mfdicts properly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39020
diff changeset
870 if self._ellipses and self._isshallow:
fbbda9ff3deb changegroup: pass mfdicts properly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39020
diff changeset
871 mfdicts = [(self._repo.manifestlog[n].read(), lr)
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
872 for (n, lr) in manifests.iteritems()]
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
873
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
874 manifests.clear()
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
875 clrevs = set(cl.rev(x) for x in clnodes)
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
876
39051
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
877 it = self.generatefiles(changedfiles, commonrevs,
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
878 source, mfdicts, fastpathlinkrev,
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
879 fnodes, clrevs)
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
880
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
881 for path, deltas in it:
39051
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
882 h = _fileheader(path)
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
883 size = len(h)
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
884 yield h
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
885
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
886 for delta in deltas:
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
887 chunks = _revisiondeltatochunks(delta, self._builddeltaheader)
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
888 for chunk in chunks:
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
889 size += len(chunk)
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
890 yield chunk
39051
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
891
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
892 close = closechunk()
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
893 size += len(close)
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
894 yield close
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
895
c4a2d19d393a changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39050
diff changeset
896 self._verbosenote(_('%8.i %s\n') % (size, path))
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
897
39040
eb8a0139ace3 changegroup: inline _close()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39039
diff changeset
898 yield closechunk()
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
899
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
900 if clnodes:
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
901 repo.hook('outgoing', node=hex(clnodes[0]), source=source)
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
902
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
903 def _generatechangelog(self, cl, nodes):
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
904 """Generate data for changelog chunks.
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
905
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
906 Returns a 2-tuple of a dict containing state and an iterable of
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
907 byte chunks. The state will not be fully populated until the
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
908 chunk stream has been fully consumed.
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
909 """
23381
cc0ff93d0c0c changegroup: fix file linkrevs during reorders (issue4462)
Durham Goode <durham@fb.com>
parents: 23226
diff changeset
910 clrevorder = {}
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
911 manifests = {}
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
912 mfl = self._repo.manifestlog
28241
a4286175ecba changegroup: drop special-casing of flat manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28240
diff changeset
913 changedfiles = set()
39036
40374b4a780f changegroup: track changelog to manifest revision map explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39035
diff changeset
914 clrevtomanifestrev = {}
19204
e9c5b1c246dc bundle-ng: move bundle generation to changegroup.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19202
diff changeset
915
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
916 # Callback for the changelog, used to collect changed files and
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
917 # manifest nodes.
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
918 # Returns the linkrev node (identity in the changelog case).
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
919 def lookupcl(x):
39265
3634ed953a42 changegroup: clean up changelog callback
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39264
diff changeset
920 c = cl.changelogrevision(x)
23381
cc0ff93d0c0c changegroup: fix file linkrevs during reorders (issue4462)
Durham Goode <durham@fb.com>
parents: 23226
diff changeset
921 clrevorder[x] = len(clrevorder)
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
922
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
923 if self._ellipses:
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
924 # Only update manifests if x is going to be sent. Otherwise we
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
925 # end up with bogus linkrevs specified for manifests and
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
926 # we skip some manifest nodes that we should otherwise
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
927 # have sent.
39034
5baafb8fe253 changegroup: rename _fullnodes to _fullclnodes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39033
diff changeset
928 if (x in self._fullclnodes
38946
ad4c4cc9a5ac changegroup: pass ellipsis roots into cgpacker constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38945
diff changeset
929 or cl.rev(x) in self._precomputedellipsis):
39265
3634ed953a42 changegroup: clean up changelog callback
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39264
diff changeset
930
3634ed953a42 changegroup: clean up changelog callback
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39264
diff changeset
931 manifestnode = c.manifest
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
932 # Record the first changeset introducing this manifest
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
933 # version.
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
934 manifests.setdefault(manifestnode, x)
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
935 # Set this narrow-specific dict so we have the lowest
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
936 # manifest revnum to look up for this cl revnum. (Part of
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
937 # mapping changelog ellipsis parents to manifest ellipsis
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
938 # parents)
39265
3634ed953a42 changegroup: clean up changelog callback
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39264
diff changeset
939 clrevtomanifestrev.setdefault(
3634ed953a42 changegroup: clean up changelog callback
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39264
diff changeset
940 cl.rev(x), mfl.rev(manifestnode))
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
941 # We can't trust the changed files list in the changeset if the
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
942 # client requested a shallow clone.
38943
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
943 if self._isshallow:
39265
3634ed953a42 changegroup: clean up changelog callback
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39264
diff changeset
944 changedfiles.update(mfl[c.manifest].read().keys())
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
945 else:
39265
3634ed953a42 changegroup: clean up changelog callback
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39264
diff changeset
946 changedfiles.update(c.files)
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
947 else:
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
948 # record the first changeset introducing this manifest version
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
949 manifests.setdefault(c.manifest, x)
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
950 # Record a complete list of potentially-changed files in
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
951 # this manifest.
39265
3634ed953a42 changegroup: clean up changelog callback
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39264
diff changeset
952 changedfiles.update(c.files)
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
953
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
954 return x
19204
e9c5b1c246dc bundle-ng: move bundle generation to changegroup.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19202
diff changeset
955
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
956 state = {
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
957 'clrevorder': clrevorder,
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
958 'manifests': manifests,
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
959 'changedfiles': changedfiles,
39036
40374b4a780f changegroup: track changelog to manifest revision map explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39035
diff changeset
960 'clrevtomanifestrev': clrevtomanifestrev,
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
961 }
38929
d706c77449f9 changegroup: move generate() modifications from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38928
diff changeset
962
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
963 gen = deltagroup(
39257
2646b8d66b7b changegroup: move node sorting into deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39214
diff changeset
964 self._repo, cl, nodes, True, lookupcl,
39055
ef3d3a2f9aa5 changegroup: refactor delta parent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39054
diff changeset
965 self._forcedeltaparentprev,
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
966 ellipses=self._ellipses,
39267
0617a700ef7b changegroup: change topics during generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39266
diff changeset
967 topic=_('changesets'),
39047
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
968 clrevtolocalrev={},
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
969 fullclnodes=self._fullclnodes,
9e8eb2b444e5 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39046
diff changeset
970 precomputedellipsis=self._precomputedellipsis)
28227
1c36cc8e7870 changegroup: extract generatemanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
971
39014
f7228c907ef4 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39013
diff changeset
972 return state, gen
28227
1c36cc8e7870 changegroup: extract generatemanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
973
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
974 def generatemanifests(self, commonrevs, clrevorder, fastpathlinkrev,
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
975 manifests, fnodes, source, clrevtolocalrev):
34161
75cc1f1e11f2 changegroup: add source parameter to generatemanifests
Durham Goode <durham@fb.com>
parents: 34160
diff changeset
976 """Returns an iterator of changegroup chunks containing manifests.
75cc1f1e11f2 changegroup: add source parameter to generatemanifests
Durham Goode <durham@fb.com>
parents: 34160
diff changeset
977
75cc1f1e11f2 changegroup: add source parameter to generatemanifests
Durham Goode <durham@fb.com>
parents: 34160
diff changeset
978 `source` is unused here, but is used by extensions like remotefilelog to
75cc1f1e11f2 changegroup: add source parameter to generatemanifests
Durham Goode <durham@fb.com>
parents: 34160
diff changeset
979 change what is sent based in pulls vs pushes, etc.
75cc1f1e11f2 changegroup: add source parameter to generatemanifests
Durham Goode <durham@fb.com>
parents: 34160
diff changeset
980 """
28227
1c36cc8e7870 changegroup: extract generatemanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
981 repo = self._repo
30308
bce79dfcf5e4 manifest: get rid of manifest.readshallowfast
Durham Goode <durham@fb.com>
parents: 30282
diff changeset
982 mfl = repo.manifestlog
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
983 tmfnodes = {'': manifests}
28227
1c36cc8e7870 changegroup: extract generatemanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27953
diff changeset
984
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
985 # Callback for the manifest, used to collect linkrevs for filelog
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
986 # revisions.
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
987 # Returns the linkrev node (collected in lookupcl).
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
988 def makelookupmflinknode(tree, nodes):
28231
3faba927dd93 changegroup: introduce makelookupmflinknode(dir)
Martin von Zweigbergk <martinvonz@google.com>
parents: 28230
diff changeset
989 if fastpathlinkrev:
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
990 assert not tree
39266
cc957b9dc335 changegroup: rename mfs to manifests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39265
diff changeset
991 return manifests.__getitem__
28231
3faba927dd93 changegroup: introduce makelookupmflinknode(dir)
Martin von Zweigbergk <martinvonz@google.com>
parents: 28230
diff changeset
992
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
993 def lookupmflinknode(x):
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
994 """Callback for looking up the linknode for manifests.
27219
beb60a898dd0 changegroup: document manifest linkrev callback some more
Augie Fackler <augie@google.com>
parents: 27218
diff changeset
995
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
996 Returns the linkrev node for the specified manifest.
27219
beb60a898dd0 changegroup: document manifest linkrev callback some more
Augie Fackler <augie@google.com>
parents: 27218
diff changeset
997
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
998 SIDE EFFECT:
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
999
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
1000 1) fclnodes gets populated with the list of relevant
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
1001 file nodes if we're not using fastpathlinkrev
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
1002 2) When treemanifests are in use, collects treemanifest nodes
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
1003 to send
27219
beb60a898dd0 changegroup: document manifest linkrev callback some more
Augie Fackler <augie@google.com>
parents: 27218
diff changeset
1004
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
1005 Note that this means manifests must be completely sent to
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
1006 the client before you can trust the list of files and
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
1007 treemanifests to send.
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
1008 """
35051
d80380ba8e7d changegroup: use any node, not min(), in treemanifest's generatemanifests
Kyle Lippincott <spectral@google.com>
parents: 34734
diff changeset
1009 clnode = nodes[x]
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1010 mdata = mfl.get(tree, x).readfast(shallow=True)
28241
a4286175ecba changegroup: drop special-casing of flat manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28240
diff changeset
1011 for p, n, fl in mdata.iterentries():
a4286175ecba changegroup: drop special-casing of flat manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28240
diff changeset
1012 if fl == 't': # subdirectory manifest
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1013 subtree = tree + p + '/'
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1014 tmfclnodes = tmfnodes.setdefault(subtree, {})
28241
a4286175ecba changegroup: drop special-casing of flat manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28240
diff changeset
1015 tmfclnode = tmfclnodes.setdefault(n, clnode)
a4286175ecba changegroup: drop special-casing of flat manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28240
diff changeset
1016 if clrevorder[clnode] < clrevorder[tmfclnode]:
a4286175ecba changegroup: drop special-casing of flat manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28240
diff changeset
1017 tmfclnodes[n] = clnode
a4286175ecba changegroup: drop special-casing of flat manifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 28240
diff changeset
1018 else:
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1019 f = tree + p
28240
1ac8ce137377 changegroup: fix treemanifests on merges
Martin von Zweigbergk <martinvonz@google.com>
parents: 28232
diff changeset
1020 fclnodes = fnodes.setdefault(f, {})
1ac8ce137377 changegroup: fix treemanifests on merges
Martin von Zweigbergk <martinvonz@google.com>
parents: 28232
diff changeset
1021 fclnode = fclnodes.setdefault(n, clnode)
1ac8ce137377 changegroup: fix treemanifests on merges
Martin von Zweigbergk <martinvonz@google.com>
parents: 28232
diff changeset
1022 if clrevorder[clnode] < clrevorder[fclnode]:
1ac8ce137377 changegroup: fix treemanifests on merges
Martin von Zweigbergk <martinvonz@google.com>
parents: 28232
diff changeset
1023 fclnodes[n] = clnode
27239
65c47779bcb5 changegroup: remove one special case from lookupmflinknode
Augie Fackler <augie@google.com>
parents: 27238
diff changeset
1024 return clnode
28231
3faba927dd93 changegroup: introduce makelookupmflinknode(dir)
Martin von Zweigbergk <martinvonz@google.com>
parents: 28230
diff changeset
1025 return lookupmflinknode
19206
6308896b1d4a bundle-ng: simplify bundle10.generate
Sune Foldager <cryo@cyanite.org>
parents: 19204
diff changeset
1026
28232
829d369fc5a8 changegroup: write root manifests and subdir manifests in a single loop
Martin von Zweigbergk <martinvonz@google.com>
parents: 28231
diff changeset
1027 while tmfnodes:
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1028 tree, nodes = tmfnodes.popitem()
39272
73cf21b2e8a6 manifest: add getstorage() to manifestlog and use it globally
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39271
diff changeset
1029 store = mfl.getstorage(tree)
39045
39f5c7afdc25 changegroup: inline _prune() into call sites
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39044
diff changeset
1030
39342
57c3864f3aad manifest: make tree a public attribute
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39272
diff changeset
1031 if not self._filematcher.visitdir(store.tree[:-1] or '.'):
39749
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1032 # No nodes to send because this directory is out of
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1033 # the client's view of the repository (probably
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1034 # because of narrow clones).
39045
39f5c7afdc25 changegroup: inline _prune() into call sites
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39044
diff changeset
1035 prunednodes = []
39f5c7afdc25 changegroup: inline _prune() into call sites
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39044
diff changeset
1036 else:
39749
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1037 # Avoid sending any manifest nodes we can prove the
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1038 # client already has by checking linkrevs. See the
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1039 # related comment in generatefiles().
39748
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1040 prunednodes = self._prunemanifests(store, nodes, commonrevs)
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1041 if tree and not prunednodes:
39043
d56a6b78de3b changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39042
diff changeset
1042 continue
d56a6b78de3b changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39042
diff changeset
1043
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1044 lookupfn = makelookupmflinknode(tree, nodes)
39020
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
1045
39052
d662959dc881 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39051
diff changeset
1046 deltas = deltagroup(
39257
2646b8d66b7b changegroup: move node sorting into deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39214
diff changeset
1047 self._repo, store, prunednodes, False, lookupfn,
39866
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
1048 self._forcedeltaparentprev,
39046
8c84f1ef949e changegroup: pass all state into group()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39045
diff changeset
1049 ellipses=self._ellipses,
39267
0617a700ef7b changegroup: change topics during generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39266
diff changeset
1050 topic=_('manifests'),
39046
8c84f1ef949e changegroup: pass all state into group()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39045
diff changeset
1051 clrevtolocalrev=clrevtolocalrev,
8c84f1ef949e changegroup: pass all state into group()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39045
diff changeset
1052 fullclnodes=self._fullclnodes,
8c84f1ef949e changegroup: pass all state into group()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39045
diff changeset
1053 precomputedellipsis=self._precomputedellipsis)
8c84f1ef949e changegroup: pass all state into group()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39045
diff changeset
1054
39261
8b9b93bf70b1 changegroup: rename dir to tree to avoid shadowing a built-in
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
1055 yield tree, deltas
39048
fcdab6629dde changegroup: emit delta group close chunk outside of deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39047
diff changeset
1056
39748
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1057 def _prunemanifests(self, store, nodes, commonrevs):
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1058 # This is split out as a separate method to allow filtering
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1059 # commonrevs in extension code.
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1060 #
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1061 # TODO(augie): this shouldn't be required, instead we should
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1062 # make filtering of revisions to send delegated to the store
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1063 # layer.
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1064 frev, flr = store.rev, store.linkrev
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1065 return [n for n in nodes if flr(frev(n)) not in commonrevs]
e03c1a63155c changegroup: tease out a temporary prune method for manifests
Augie Fackler <augie@google.com>
parents: 39747
diff changeset
1066
24897
5c35a6040352 changegroup: document that 'source' parameter exists for extensions
Martin von Zweigbergk <martinvonz@google.com>
parents: 24896
diff changeset
1067 # The 'source' parameter is useful for extensions
39037
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1068 def generatefiles(self, changedfiles, commonrevs, source,
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1069 mfdicts, fastpathlinkrev, fnodes, clrevs):
38928
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1070 changedfiles = list(filter(self._filematcher, changedfiles))
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1071
39037
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1072 if not fastpathlinkrev:
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1073 def normallinknodes(unused, fname):
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1074 return fnodes.get(fname, {})
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1075 else:
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1076 cln = self._repo.changelog.node
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1077
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1078 def normallinknodes(store, fname):
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1079 flinkrev = store.linkrev
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1080 fnode = store.node
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1081 revs = ((r, flinkrev(r)) for r in store)
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1082 return dict((fnode(r), cln(lr))
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1083 for r, lr in revs if lr in clrevs)
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1084
39039
a6e1ff40e335 changegroup: pass clrevtolocalrev to each group
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39038
diff changeset
1085 clrevtolocalrev = {}
a6e1ff40e335 changegroup: pass clrevtolocalrev to each group
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39038
diff changeset
1086
38943
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
1087 if self._isshallow:
38928
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1088 # In a shallow clone, the linknodes callback needs to also include
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1089 # those file nodes that are in the manifests we sent but weren't
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1090 # introduced by those manifests.
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1091 commonctxs = [self._repo[c] for c in commonrevs]
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1092 clrev = self._repo.changelog.rev
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1093
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1094 def linknodes(flog, fname):
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1095 for c in commonctxs:
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1096 try:
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1097 fnode = c.filenode(fname)
39039
a6e1ff40e335 changegroup: pass clrevtolocalrev to each group
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39038
diff changeset
1098 clrevtolocalrev[c.rev()] = flog.rev(fnode)
38928
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1099 except error.ManifestLookupError:
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1100 pass
39037
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1101 links = normallinknodes(flog, fname)
38928
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1102 if len(links) != len(mfdicts):
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1103 for mf, lr in mfdicts:
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1104 fnode = mf.get(fname, None)
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1105 if fnode in links:
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1106 links[fnode] = min(links[fnode], lr, key=clrev)
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1107 elif fnode:
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1108 links[fnode] = lr
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1109 return links
39037
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1110 else:
b83e9c503f2f changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39036
diff changeset
1111 linknodes = normallinknodes
38928
a06aab274aef changegroup: move generatefiles() from narrow
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38927
diff changeset
1112
19334
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
1113 repo = self._repo
39267
0617a700ef7b changegroup: change topics during generation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39266
diff changeset
1114 progress = repo.ui.makeprogress(_('files'), unit=_('files'),
38416
1c5c4a5dd86d changegroup: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38388
diff changeset
1115 total=len(changedfiles))
19334
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
1116 for i, fname in enumerate(sorted(changedfiles)):
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
1117 filerevlog = repo.file(fname)
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
1118 if not filerevlog:
37339
5859800edfc5 changegroup: remove "revlog" from error message
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37135
diff changeset
1119 raise error.Abort(_("empty or missing file data for %s") %
5859800edfc5 changegroup: remove "revlog" from error message
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37135
diff changeset
1120 fname)
19334
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
1121
39039
a6e1ff40e335 changegroup: pass clrevtolocalrev to each group
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39038
diff changeset
1122 clrevtolocalrev.clear()
a6e1ff40e335 changegroup: pass clrevtolocalrev to each group
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39038
diff changeset
1123
19334
95a49112e7ab bundle: move file chunk generation to it's own function
Durham Goode <durham@fb.com>
parents: 19325
diff changeset
1124 linkrevnodes = linknodes(filerevlog, fname)
19207
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
1125 # Lookup for filenodes, we collected the linkrev nodes above in the
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
1126 # fastpath case and with lookupmf in the slowpath case.
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
1127 def lookupfilelog(x):
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
1128 return linkrevnodes[x]
a67e1380dfbd bundle-ng: simplify lookup and state handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 19206
diff changeset
1129
39045
39f5c7afdc25 changegroup: inline _prune() into call sites
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39044
diff changeset
1130 frev, flr = filerevlog.rev, filerevlog.linkrev
39749
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1131 # Skip sending any filenode we know the client already
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1132 # has. This avoids over-sending files relatively
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1133 # inexpensively, so it's not a problem if we under-filter
5adc5fe41a7d changegroup: reintroduce some comments that have gotten lost over the years
Augie Fackler <augie@google.com>
parents: 39748
diff changeset
1134 # here.
39045
39f5c7afdc25 changegroup: inline _prune() into call sites
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39044
diff changeset
1135 filenodes = [n for n in linkrevnodes
39f5c7afdc25 changegroup: inline _prune() into call sites
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39044
diff changeset
1136 if flr(frev(n)) not in commonrevs]
39f5c7afdc25 changegroup: inline _prune() into call sites
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39044
diff changeset
1137
39058
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1138 if not filenodes:
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1139 continue
39020
037debbf869c changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39019
diff changeset
1140
39058
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1141 progress.update(i + 1, item=fname)
39046
8c84f1ef949e changegroup: pass all state into group()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39045
diff changeset
1142
39058
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1143 deltas = deltagroup(
39257
2646b8d66b7b changegroup: move node sorting into deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39214
diff changeset
1144 self._repo, filerevlog, filenodes, False, lookupfilelog,
39866
db5501d93bcf changegroup: remove reordering control (BC)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39749
diff changeset
1145 self._forcedeltaparentprev,
39058
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1146 ellipses=self._ellipses,
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1147 clrevtolocalrev=clrevtolocalrev,
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1148 fullclnodes=self._fullclnodes,
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1149 precomputedellipsis=self._precomputedellipsis)
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1150
0b5f534df82a changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39057
diff changeset
1151 yield fname, deltas
39048
fcdab6629dde changegroup: emit delta group close chunk outside of deltagroup()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39047
diff changeset
1152
38416
1c5c4a5dd86d changegroup: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38388
diff changeset
1153 progress.complete()
19200
4cfdec944edf bundle-ng: move group into the bundler
Sune Foldager <cryo@cyanite.org>
parents: 19199
diff changeset
1154
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1155 def _makecg1packer(repo, filematcher, bundlecaps, ellipses=False,
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1156 shallow=False, ellipsisroots=None, fullnodes=None):
38936
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
1157 builddeltaheader = lambda d: _CHANGEGROUPV1_DELTA_HEADER.pack(
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
1158 d.node, d.p1node, d.p2node, d.linknode)
27432
77d25b913f80 changegroup: introduce cg3, which has support for exchanging treemanifests
Augie Fackler <augie@google.com>
parents: 27241
diff changeset
1159
38941
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
1160 return cgpacker(repo, filematcher, b'01',
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
1161 builddeltaheader=builddeltaheader,
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
1162 manifestsend=b'',
39055
ef3d3a2f9aa5 changegroup: refactor delta parent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39054
diff changeset
1163 forcedeltaparentprev=True,
38943
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
1164 bundlecaps=bundlecaps,
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1165 ellipses=ellipses,
38946
ad4c4cc9a5ac changegroup: pass ellipsis roots into cgpacker constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38945
diff changeset
1166 shallow=shallow,
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1167 ellipsisroots=ellipsisroots,
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1168 fullnodes=fullnodes)
38933
19344024a8e1 changegroup: define functions for creating changegroup packers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38932
diff changeset
1169
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1170 def _makecg2packer(repo, filematcher, bundlecaps, ellipses=False,
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1171 shallow=False, ellipsisroots=None, fullnodes=None):
38936
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
1172 builddeltaheader = lambda d: _CHANGEGROUPV2_DELTA_HEADER.pack(
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
1173 d.node, d.p1node, d.p2node, d.basenode, d.linknode)
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
1174
38941
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
1175 return cgpacker(repo, filematcher, b'02',
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
1176 builddeltaheader=builddeltaheader,
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
1177 manifestsend=b'',
38943
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
1178 bundlecaps=bundlecaps,
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1179 ellipses=ellipses,
38946
ad4c4cc9a5ac changegroup: pass ellipsis roots into cgpacker constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38945
diff changeset
1180 shallow=shallow,
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1181 ellipsisroots=ellipsisroots,
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1182 fullnodes=fullnodes)
38933
19344024a8e1 changegroup: define functions for creating changegroup packers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38932
diff changeset
1183
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1184 def _makecg3packer(repo, filematcher, bundlecaps, ellipses=False,
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1185 shallow=False, ellipsisroots=None, fullnodes=None):
38936
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
1186 builddeltaheader = lambda d: _CHANGEGROUPV3_DELTA_HEADER.pack(
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
1187 d.node, d.p1node, d.p2node, d.basenode, d.linknode, d.flags)
bd64b8b8f0dd changegroup: pass function to build delta header into constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38935
diff changeset
1188
38941
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
1189 return cgpacker(repo, filematcher, b'03',
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
1190 builddeltaheader=builddeltaheader,
4c99c6d1ef95 changegroup: rename cg1packer to cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38940
diff changeset
1191 manifestsend=closechunk(),
38943
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
1192 bundlecaps=bundlecaps,
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1193 ellipses=ellipses,
38946
ad4c4cc9a5ac changegroup: pass ellipsis roots into cgpacker constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38945
diff changeset
1194 shallow=shallow,
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1195 ellipsisroots=ellipsisroots,
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1196 fullnodes=fullnodes)
38933
19344024a8e1 changegroup: define functions for creating changegroup packers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38932
diff changeset
1197
19344024a8e1 changegroup: define functions for creating changegroup packers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38932
diff changeset
1198 _packermap = {'01': (_makecg1packer, cg1unpacker),
26709
42733e956887 changegroup: reformat packermap and add comment
Augie Fackler <augie@google.com>
parents: 26708
diff changeset
1199 # cg2 adds support for exchanging generaldelta
38933
19344024a8e1 changegroup: define functions for creating changegroup packers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38932
diff changeset
1200 '02': (_makecg2packer, cg2unpacker),
27753
d4071cc73f46 changegroup3: add empty chunk separating directories and files
Martin von Zweigbergk <martinvonz@google.com>
parents: 27752
diff changeset
1201 # cg3 adds support for exchanging revlog flags and treemanifests
38933
19344024a8e1 changegroup: define functions for creating changegroup packers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38932
diff changeset
1202 '03': (_makecg3packer, cg3unpacker),
26709
42733e956887 changegroup: reformat packermap and add comment
Augie Fackler <augie@google.com>
parents: 26708
diff changeset
1203 }
23168
a92ba36a1a9d changegroup: add a "packermap" dictionary to track different packer versions
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 22971
diff changeset
1204
30632
7ace5304fec5 changegroup: pass 'repo' to allsupportedversions
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30631
diff changeset
1205 def allsupportedversions(repo):
27928
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
1206 versions = set(_packermap.keys())
30632
7ace5304fec5 changegroup: pass 'repo' to allsupportedversions
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30631
diff changeset
1207 if not (repo.ui.configbool('experimental', 'changegroup3') or
30633
a001cd7296a5 changegroup: simplify logic around enabling changegroup 03
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30632
diff changeset
1208 repo.ui.configbool('experimental', 'treemanifest') or
a001cd7296a5 changegroup: simplify logic around enabling changegroup 03
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30632
diff changeset
1209 'treemanifest' in repo.requirements):
30631
438532c99b54 changegroup: simplify 'allsupportedversions' logic
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30594
diff changeset
1210 versions.discard('03')
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1211 return versions
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1212
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1213 # Changegroup versions that can be applied to the repo
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1214 def supportedincomingversions(repo):
30633
a001cd7296a5 changegroup: simplify logic around enabling changegroup 03
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30632
diff changeset
1215 return allsupportedversions(repo)
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1216
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1217 # Changegroup versions that can be created from the repo
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1218 def supportedoutgoingversions(repo):
30632
7ace5304fec5 changegroup: pass 'repo' to allsupportedversions
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 30631
diff changeset
1219 versions = allsupportedversions(repo)
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1220 if 'treemanifest' in repo.requirements:
27928
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
1221 # Versions 01 and 02 support only flat manifests and it's just too
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
1222 # expensive to convert between the flat manifest and tree manifest on
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
1223 # the fly. Since tree manifests are hashed differently, all of history
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
1224 # would have to be converted. Instead, we simply don't even pretend to
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
1225 # support versions 01 and 02.
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
1226 versions.discard('01')
c0f11347b107 changegroup: don't support versions 01 and 02 with treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27920
diff changeset
1227 versions.discard('02')
38874
a232e6744ba3 narrow: move requirement constant from changegroup to repository
Martin von Zweigbergk <martinvonz@google.com>
parents: 38846
diff changeset
1228 if repository.NARROW_REQUIREMENT in repo.requirements:
36495
94709406f10d narrow: move changegroup.supportedoutgoingversions() override to core
Martin von Zweigbergk <martinvonz@google.com>
parents: 36494
diff changeset
1229 # Versions 01 and 02 don't support revlog flags, and we need to
94709406f10d narrow: move changegroup.supportedoutgoingversions() override to core
Martin von Zweigbergk <martinvonz@google.com>
parents: 36494
diff changeset
1230 # support that for stripping and unbundling to work.
94709406f10d narrow: move changegroup.supportedoutgoingversions() override to core
Martin von Zweigbergk <martinvonz@google.com>
parents: 36494
diff changeset
1231 versions.discard('01')
94709406f10d narrow: move changegroup.supportedoutgoingversions() override to core
Martin von Zweigbergk <martinvonz@google.com>
parents: 36494
diff changeset
1232 versions.discard('02')
37135
a54113fcc8c9 lfs: move the 'supportedoutgoingversions' handling to changegroup.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 37087
diff changeset
1233 if LFS_REQUIREMENT in repo.requirements:
a54113fcc8c9 lfs: move the 'supportedoutgoingversions' handling to changegroup.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 37087
diff changeset
1234 # Versions 01 and 02 don't support revlog flags, and we need to
a54113fcc8c9 lfs: move the 'supportedoutgoingversions' handling to changegroup.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 37087
diff changeset
1235 # mark LFS entries with REVIDX_EXTSTORED.
a54113fcc8c9 lfs: move the 'supportedoutgoingversions' handling to changegroup.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 37087
diff changeset
1236 versions.discard('01')
a54113fcc8c9 lfs: move the 'supportedoutgoingversions' handling to changegroup.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 37087
diff changeset
1237 versions.discard('02')
a54113fcc8c9 lfs: move the 'supportedoutgoingversions' handling to changegroup.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 37087
diff changeset
1238
27752
29cfc474c5fd changegroup3: introduce experimental.changegroup3 boolean config
Martin von Zweigbergk <martinvonz@google.com>
parents: 27751
diff changeset
1239 return versions
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
1240
33676
91f0677dc920 repair: preserve phase also when not using generaldelta (issue5678)
Martin von Zweigbergk <martinvonz@google.com>
parents: 33461
diff changeset
1241 def localversion(repo):
91f0677dc920 repair: preserve phase also when not using generaldelta (issue5678)
Martin von Zweigbergk <martinvonz@google.com>
parents: 33461
diff changeset
1242 # Finds the best version to use for bundles that are meant to be used
91f0677dc920 repair: preserve phase also when not using generaldelta (issue5678)
Martin von Zweigbergk <martinvonz@google.com>
parents: 33461
diff changeset
1243 # locally, such as those from strip and shelve, and temporary bundles.
91f0677dc920 repair: preserve phase also when not using generaldelta (issue5678)
Martin von Zweigbergk <martinvonz@google.com>
parents: 33461
diff changeset
1244 return max(supportedoutgoingversions(repo))
91f0677dc920 repair: preserve phase also when not using generaldelta (issue5678)
Martin von Zweigbergk <martinvonz@google.com>
parents: 33461
diff changeset
1245
27929
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
1246 def safeversion(repo):
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
1247 # Finds the smallest version that it's safe to assume clients of the repo
27931
1289a122cf3f shelve: use cg3 for treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27929
diff changeset
1248 # will support. For example, all hg versions that support generaldelta also
1289a122cf3f shelve: use cg3 for treemanifests
Martin von Zweigbergk <martinvonz@google.com>
parents: 27929
diff changeset
1249 # support changegroup 02.
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1250 versions = supportedoutgoingversions(repo)
27929
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
1251 if 'generaldelta' in repo.requirements:
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
1252 versions.discard('01')
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
1253 assert versions
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
1254 return min(versions)
3b2ac2115464 changegroup: introduce safeversion()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27928
diff changeset
1255
38943
cdb9bc216771 changegroup: declare shallow flag in constructor
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38942
diff changeset
1256 def getbundler(version, repo, bundlecaps=None, filematcher=None,
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1257 ellipses=False, shallow=False, ellipsisroots=None,
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1258 fullnodes=None):
27953
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066)
Martin von Zweigbergk <martinvonz@google.com>
parents: 27946
diff changeset
1259 assert version in supportedoutgoingversions(repo)
38834
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1260
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1261 if filematcher is None:
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1262 filematcher = matchmod.alwaysmatcher(repo.root, '')
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1263
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1264 if version == '01' and not filematcher.always():
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1265 raise error.ProgrammingError('version 01 changegroups do not support '
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1266 'sparse file matchers')
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1267
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1268 if ellipses and version in (b'01', b'02'):
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1269 raise error.Abort(
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1270 _('ellipsis nodes require at least cg3 on client and server, '
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1271 'but negotiated version %s') % version)
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1272
38834
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1273 # Requested files could include files not in the local store. So
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1274 # filter those out.
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1275 filematcher = matchmod.intersectmatchers(repo.narrowmatch(),
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1276 filematcher)
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1277
38933
19344024a8e1 changegroup: define functions for creating changegroup packers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38932
diff changeset
1278 fn = _packermap[version][0]
38947
1469584ad5fe changegroup: specify ellipses mode explicitly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38946
diff changeset
1279 return fn(repo, filematcher, bundlecaps, ellipses=ellipses,
38948
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1280 shallow=shallow, ellipsisroots=ellipsisroots,
1af339c22aeb changegroup: move fullnodes into cgpacker
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38947
diff changeset
1281 fullnodes=fullnodes)
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
1282
29593
953839de96ab bundle2: store changeset count when creating file bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29371
diff changeset
1283 def getunbundler(version, fh, alg, extras=None):
953839de96ab bundle2: store changeset count when creating file bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29371
diff changeset
1284 return _packermap[version][1](fh, alg, extras=extras)
27751
a40e2f7fe49d changegroup: hide packermap behind methods
Martin von Zweigbergk <martinvonz@google.com>
parents: 27739
diff changeset
1285
20926
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
1286 def _changegroupinfo(repo, nodes, source):
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
1287 if repo.ui.verbose or source == 'bundle':
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
1288 repo.ui.status(_("%d changesets found\n") % len(nodes))
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
1289 if repo.ui.debugflag:
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
1290 repo.ui.debug("list of changesets:\n")
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
1291 for node in nodes:
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
1292 repo.ui.debug("%s\n" % hex(node))
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
1293
34111
f85dfde1731a changegroup: replace getsubset with makechangegroup
Durham Goode <durham@fb.com>
parents: 34106
diff changeset
1294 def makechangegroup(repo, outgoing, version, source, fastpath=False,
f85dfde1731a changegroup: replace getsubset with makechangegroup
Durham Goode <durham@fb.com>
parents: 34106
diff changeset
1295 bundlecaps=None):
f85dfde1731a changegroup: replace getsubset with makechangegroup
Durham Goode <durham@fb.com>
parents: 34106
diff changeset
1296 cgstream = makestream(repo, outgoing, version, source,
f85dfde1731a changegroup: replace getsubset with makechangegroup
Durham Goode <durham@fb.com>
parents: 34106
diff changeset
1297 fastpath=fastpath, bundlecaps=bundlecaps)
f85dfde1731a changegroup: replace getsubset with makechangegroup
Durham Goode <durham@fb.com>
parents: 34106
diff changeset
1298 return getunbundler(version, util.chunkbuffer(cgstream), None,
f85dfde1731a changegroup: replace getsubset with makechangegroup
Durham Goode <durham@fb.com>
parents: 34106
diff changeset
1299 {'clcount': len(outgoing.missing) })
f85dfde1731a changegroup: replace getsubset with makechangegroup
Durham Goode <durham@fb.com>
parents: 34106
diff changeset
1300
34118
92f1e2be8ab6 changegroup: rename getsubsetraw to makestream
Durham Goode <durham@fb.com>
parents: 34116
diff changeset
1301 def makestream(repo, outgoing, version, source, fastpath=False,
38834
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1302 bundlecaps=None, filematcher=None):
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1303 bundler = getbundler(version, repo, bundlecaps=bundlecaps,
1d01cf0416a5 changegroup: move file matcher from narrow extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1304 filematcher=filematcher)
34118
92f1e2be8ab6 changegroup: rename getsubsetraw to makestream
Durham Goode <durham@fb.com>
parents: 34116
diff changeset
1305
20925
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1306 repo = repo.unfiltered()
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1307 commonrevs = outgoing.common
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1308 csets = outgoing.missing
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1309 heads = outgoing.missingheads
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1310 # We go through the fast path if we get told to, or if all (unfiltered
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1311 # heads have been requested (since we then know there all linkrevs will
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1312 # be pulled by the client).
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1313 heads.sort()
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1314 fastpathlinkrev = fastpath or (
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1315 repo.filtername is None and heads == sorted(repo.heads()))
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1316
5174c48ed8d8 localrepo: move the _changegroupsubset method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20675
diff changeset
1317 repo.hook('preoutgoing', throw=True, source=source)
20926
7c1ed40e3325 localrepo: move the changegroupinfo method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20925
diff changeset
1318 _changegroupinfo(repo, csets, source)
23177
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
1319 return bundler.generate(commonrevs, csets, fastpathlinkrev, source)
706547a14b8b changegroup: introduce "raw" versions of some commands
Sune Foldager <cryo@cyanite.org>
parents: 23168
diff changeset
1320
28361
277a22cd8741 changegroup: progress for added files is not measured in "chunks"
Martin von Zweigbergk <martinvonz@google.com>
parents: 28360
diff changeset
1321 def _addchangegroupfiles(repo, source, revmap, trp, expectedfiles, needfiles):
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1322 revisions = 0
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1323 files = 0
38388
daa08d45740f changegroup: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38379
diff changeset
1324 progress = repo.ui.makeprogress(_('files'), unit=_('files'),
daa08d45740f changegroup: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38379
diff changeset
1325 total=expectedfiles)
29736
4e7be6e33269 changegroup: use `iter(callable, sentinel)` instead of while True
Augie Fackler <augie@google.com>
parents: 29704
diff changeset
1326 for chunkdata in iter(source.filelogheader, {}):
28361
277a22cd8741 changegroup: progress for added files is not measured in "chunks"
Martin von Zweigbergk <martinvonz@google.com>
parents: 28360
diff changeset
1327 files += 1
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1328 f = chunkdata["filename"]
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1329 repo.ui.debug("adding %s revisions\n" % f)
38388
daa08d45740f changegroup: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38379
diff changeset
1330 progress.increment()
27754
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
1331 fl = repo.file(f)
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1332 o = len(fl)
24120
a450e0a2ba0a revlog: in addgroup, reject ill-formed deltas based on censored nodes
Mike Edgar <adgar@google.com>
parents: 23897
diff changeset
1333 try:
34298
1db9abf407c5 revlog: add revmap back to revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34264
diff changeset
1334 deltas = source.deltaiter()
1db9abf407c5 revlog: add revmap back to revlog.addgroup
Durham Goode <durham@fb.com>
parents: 34264
diff changeset
1335 if not fl.addgroup(deltas, revmap, trp):
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
1336 raise error.Abort(_("received file revlog group is empty"))
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25624
diff changeset
1337 except error.CensoredBaseError as e:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
1338 raise error.Abort(_("received delta base is censored: %s") % e)
27754
a09f143daaf4 changegroup3: move treemanifest support into _unpackmanifests()
Martin von Zweigbergk <martinvonz@google.com>
parents: 27753
diff changeset
1339 revisions += len(fl) - o
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1340 if f in needfiles:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1341 needs = needfiles[f]
38823
e7aa113b14f7 global: use pycompat.xrange()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38557
diff changeset
1342 for new in pycompat.xrange(o, len(fl)):
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1343 n = fl.node(new)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1344 if n in needs:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1345 needs.remove(n)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1346 else:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
1347 raise error.Abort(
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1348 _("received spurious file revlog entry"))
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1349 if not needs:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1350 del needfiles[f]
38388
daa08d45740f changegroup: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38379
diff changeset
1351 progress.complete()
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1352
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1353 for f, needs in needfiles.iteritems():
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1354 fl = repo.file(f)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1355 for n in needs:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1356 try:
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1357 fl.rev(n)
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1358 except error.LookupError:
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 26540
diff changeset
1359 raise error.Abort(
20932
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1360 _('missing file data for %s:%s - run hg verify') %
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1361 (f, hex(n)))
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1362
0ac83e4e4f7c localrepo: move the addchangegroupfiles method in changegroup module
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20931
diff changeset
1363 return revisions, files