Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/streamclone.py @ 52924:7f848cfc4286

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
stream-clone-v2: use dedicated threads to write the data on disk This could provide a significant performance boost, but various implementation details means it is currently slower. More update will help make this boost real, but here is the basic idea. The implementation in this patch is unbounded in memory which could be a problem in some situation. We will deal with that soon. There is the benchmark result showing a slower run ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog before: 0.249693 ~~~~~ after: 0.275081 (+10.17%, +0.03) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog before: 5.317709 ~~~~~ after: 6.783031 (+27.56%, +1.47) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog before: 5.398368 ~~~~~ after: 6.737864 (+24.81%, +1.34) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog before: 3.acbb55 ~~~~~ after: 3.758324 (+22.48%, +0.69) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog before: 7.244015 ~~~~~ after: 9.128669 (+26.02%, +1.88) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog before: 13.136674 ~~~~~ after: 16.374306 (+24.65%, +3.24) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog before: 52.253858 ~~~~~ after: 66.955037 (+28.13%, +14.70) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog before: 51.934795 ~~~~~ after: 66.561340 (+28.16%, +14.63) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no before: 130.584329 ~~~~~ after: 138.770454 (+6.27%, +8.19)
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Mon, 20 Jan 2025 15:13:30 +0100
parents 307c4a0b91a0
children 58baa86c7a02
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
51901
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
8 from __future__ import annotations
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
10 import contextlib
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
11 import errno
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
12 import os
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
13 import queue
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
14 import struct
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
15 import threading
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
16
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
17 from typing import (
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
18 Iterable,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
19 Iterator,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
20 Optional,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
21 Set,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
22 Tuple,
52923
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
23 Type,
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
24 )
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
25
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
26 from .i18n import _
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
27 from .interfaces import repository
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
28 from . import (
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
29 bookmarks,
50557
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
30 bundle2 as bundle2mod,
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
31 cacheutil,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
32 error,
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
33 narrowspec,
32764
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
34 phases,
38197
aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path
Yuya Nishihara <yuya@tcha.org>
parents: 37638
diff changeset
35 pycompat,
46665
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
36 requirements as requirementsmod,
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
37 scmutil,
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
38 store,
50286
3d0b5760851c undo-files: move the undo cleanup code in the transaction module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50279
diff changeset
39 transaction,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
40 util,
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
41 )
48707
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
42 from .revlogutils import (
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
43 nodemap,
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
44 )
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
45
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
46
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
47 def new_stream_clone_requirements(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
48 default_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
49 streamed_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
50 ) -> Set[bytes]:
48618
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
51 """determine the final set of requirement for a new stream clone
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
52
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
53 this method combine the "default" requirements that a new repository would
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
54 use with the constaint we get from the stream clone content. We keep local
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
55 configuration choice when possible.
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
56 """
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
57 requirements = set(default_requirements)
48623
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48622
diff changeset
58 requirements -= requirementsmod.STREAM_FIXED_REQUIREMENTS
48618
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
59 requirements.update(streamed_requirements)
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
60 return requirements
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
61
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
62
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
63 def streamed_requirements(repo) -> Set[bytes]:
48619
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
64 """the set of requirement the new clone will have to support
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
65
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
66 This is used for advertising the stream options and to generate the actual
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
67 stream content."""
48623
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48622
diff changeset
68 requiredformats = (
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48622
diff changeset
69 repo.requirements & requirementsmod.STREAM_FIXED_REQUIREMENTS
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48622
diff changeset
70 )
48619
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
71 return requiredformats
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
72
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
73
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
74 def canperformstreamclone(pullop, bundle2: bool = False):
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
75 """Whether it is possible to perform a streaming clone as part of pull.
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
76
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
77 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
78 bundle2 and only through bundle2.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
79
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
80 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
81 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
82 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
83 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
84 """
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
85 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
86 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
87
50556
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
88 # should we consider streaming clone at all ?
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
89 streamrequested = pullop.streamclonerequested
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
90 # If we don't have a preference, let the server decide for us. This
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
91 # likely only comes into play in LANs.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
92 if streamrequested is None:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
93 # The server can advertise whether to prefer streaming clone.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
94 streamrequested = remote.capable(b'stream-preferred')
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
95 if not streamrequested:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
96 return False, None
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
97
50554
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50550
diff changeset
98 # Streaming clone only works on an empty destination repository
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50550
diff changeset
99 if len(repo):
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50550
diff changeset
100 return False, None
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50550
diff changeset
101
50555
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50554
diff changeset
102 # Streaming clone only works if all data is being requested.
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50554
diff changeset
103 if pullop.heads:
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50554
diff changeset
104 return False, None
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50554
diff changeset
105
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
106 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
107 if pullop.canusebundle2:
50557
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
108 local_caps = bundle2mod.getrepocaps(repo, role=b'client')
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
109 local_supported = set(local_caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
110 remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
111 bundle2supported = bool(local_supported & remote_supported)
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
112 # else
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
113 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
114 # the versions we support. Fall back and possibly allow legacy.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
115
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
116 # Ensures legacy code path uses available bundle2.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
117 if bundle2supported and not bundle2:
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
118 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
119 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
120 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
121 return False, None
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
122
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
123 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
124 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
125 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
126 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
127 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
128 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
129 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
130 requirements = set()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
131 if remote.capable(b'stream'):
46665
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
132 requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
133 else:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
134 streamreqs = remote.capable(b'streamreqs')
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
135 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
136 if not streamreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
137 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
138 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
139 b'warning: stream clone requested but server has them '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
140 b'disabled\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
141 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
142 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
143 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
144
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
145 streamreqs = set(streamreqs.split(b','))
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
146 # Server requires something we don't support. Bail.
48359
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
147 missingreqs = streamreqs - repo.supported
32299
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30995
diff changeset
148 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30995
diff changeset
149 pullop.repo.ui.warn(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
150 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
151 b'warning: stream clone requested but client is missing '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
152 b'requirements: %s\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
153 )
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
154 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
155 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
156 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
157 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
158 b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
159 b'for more information)\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
160 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
161 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
162 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
163 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
164
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
165 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
166
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
167
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
168 def maybeperformlegacystreamclone(pullop) -> None:
26462
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
169 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
170
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
171 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
172 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
173
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
174 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
175 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
176 """
39716
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
177 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
178
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
179 supported, requirements = canperformstreamclone(pullop)
26458
362793295640 streamclone: refactor maybeperformstreamclone to take a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26447
diff changeset
180
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
181 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
182 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
183
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
184 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
185 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
186
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
187 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
188 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
189 rbranchmap = None
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
190 if remote.capable(b'branchmap'):
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
191 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
192 rbranchmap = e.callcommand(b'branchmap', {}).result()
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
193
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
194 repo.ui.status(_(b'streaming all changes\n'))
26470
4b5647d9ee13 streamclone: move "streaming all changes" message location
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26469
diff changeset
195
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
196 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
197 fp = e.callcommand(b'stream_out', {}).result()
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
198
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
199 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
200 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
201 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
202 # doesn't matter.
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
203 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
204 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
205 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
206 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
207 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
208 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
209 )
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
210 if resp == 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
211 raise error.Abort(_(b'operation forbidden by server'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
212 elif resp == 2:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
213 raise error.Abort(_(b'locking the remote repository failed'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
214 elif resp != 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
215 raise error.Abort(_(b'the server sent an unknown error code'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
216
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
217 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
218 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
219 filecount, bytecount = map(int, l.split(b' ', 1))
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
220 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
221 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
222 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
223 )
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
224
27850
49cfddbf54ba with: use context manager in maybeperformlegacystreamclone
Bryan O'Sullivan <bryano@fb.com>
parents: 27845
diff changeset
225 with repo.lock():
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
226 consumev1(repo, fp, filecount, bytecount)
48618
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
227 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
228 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
229 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
230 )
39716
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
231 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
232 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
233 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
234 scmutil.writereporequirements(repo)
48707
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
235 nodemap.post_stream_cleanup(repo)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
236
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
237 if rbranchmap:
41626
328ca3b9e545 branchmap: encapsulate cache updating in the map itself
Martijn Pieters <mj@octobus.net>
parents: 40544
diff changeset
238 repo._branchcaches.replace(repo, rbranchmap)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
239
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
240 repo.invalidate()
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
241
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
242
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
243 def allowservergeneration(repo) -> bool:
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
244 """Whether streaming clones are allowed from the server."""
40029
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39855
diff changeset
245 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39855
diff changeset
246 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39855
diff changeset
247
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
248 if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
32764
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
249 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
250
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
251 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
252 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
253 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
254 if secret:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
255 return repo.ui.configbool(b'server', b'uncompressedallowsecret')
32764
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
256
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
257 return True
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
258
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
259
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
260 # This is it's own function so extensions can override it.
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
261 def _walkstreamfiles(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
262 repo, matcher=None, phase: bool = False, obsolescence: bool = False
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
263 ):
50548
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50546
diff changeset
264 return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
265
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
266
52720
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
267 def _report_transferred(
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
268 repo, start_time: float, file_count: int, byte_count: int
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
269 ):
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
270 """common utility to report time it took to apply the stream bundle"""
52718
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
271 elapsed = util.timer() - start_time
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
272 if elapsed <= 0:
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
273 elapsed = 0.001
52720
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
274 m = _(b'stream-cloned %d files / %s in %.1f seconds (%s/sec)\n')
52718
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
275 m %= (
52720
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
276 file_count,
52718
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
277 util.bytecount(byte_count),
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
278 elapsed,
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
279 util.bytecount(byte_count / elapsed),
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
280 )
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
281 repo.ui.status(m)
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
282
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52714
diff changeset
283
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
284 def generatev1(repo) -> tuple[int, int, Iterator[bytes]]:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
285 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
286
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
287 This returns a 3-tuple of (file count, byte size, data iterator).
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
288
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
289 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
290 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
291 delimited by a null byte.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
292
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
293 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
294 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
295
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
296 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
297 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
298 for adding it.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
299
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
300 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
301 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
302 """
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
303 entries = []
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
304 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
305 # Get consistent snapshot of repo, lock during scan.
27845
7417e1c10253 with: use context manager in streamclone generatev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27794
diff changeset
306 with repo.lock():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
307 repo.ui.debug(b'scanning\n')
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
308 _test_sync_point_walk_1_2(repo)
50505
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50288
diff changeset
309 for entry in _walkstreamfiles(repo):
50506
9fdc28e21b68 store: introduce a EntryFile object to actually access file info
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50505
diff changeset
310 for f in entry.files():
50511
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50506
diff changeset
311 file_size = f.file_size(repo.store.vfs)
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50506
diff changeset
312 if file_size:
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50506
diff changeset
313 entries.append((f.unencoded_path, file_size))
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50506
diff changeset
314 total_bytes += file_size
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
315 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
316 _test_sync_point_walk_4(repo)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
317
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
318 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
319 b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
320 )
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
321
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
322 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
323 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
324
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
325 def emitrevlogdata() -> Iterator[bytes]:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
326 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
327 if debugflag:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
328 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
329 # partially encode name over the wire for backwards compat
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
330 yield b'%s\0%d\n' % (store.encodedir(name), size)
33411
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
331 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
332 # trusted by the local repo) and expensive
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
333 with svfs(name, b'rb', auditpath=False) as fp:
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
334 if size <= 65536:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
335 yield fp.read(size)
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
336 else:
52669
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
337 yield from util.filechunkiter(fp, limit=size)
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
338
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
339 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
340
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
341
52709
279e217d6041 typing: lock in the new type annotations detected with the pyupgrade changes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52669
diff changeset
342 def generatev1wireproto(repo) -> Iterator[bytes]:
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
343 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
344
35495
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
345 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
346 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
347 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
348
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
349 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
350 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
351 a permissions error for the server process).
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
352 """
35495
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
353 if not allowservergeneration(repo):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
354 yield b'1\n'
35495
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
355 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
356
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
357 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
358 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
359 except error.LockError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
360 yield b'2\n'
35495
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
361 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
362
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
363 # Indicates successful response.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
364 yield b'0\n'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
365 yield b'%d %d\n' % (filecount, bytecount)
52669
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
366 yield from it
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
367
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
368
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
369 def generatebundlev1(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
370 repo, compression: bytes = b'UN'
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
371 ) -> tuple[Set[bytes], Iterator[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
372 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
373
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
374 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
375 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
376
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
377 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
378 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
379
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
380 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
381 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
382
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
383 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
384 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
385 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
386 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
387
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
388 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
389 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
390
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
391 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
392 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
393 if compression != b'UN':
52714
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52713
diff changeset
394 raise ValueError('we do not support the compression argument yet')
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
395
48619
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
396 requirements = streamed_requirements(repo)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
397 requires = b','.join(sorted(requirements))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
398
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
399 def gen() -> Iterator[bytes]:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
400 yield b'HGS1'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
401 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
402
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
403 filecount, bytecount, it = generatev1(repo)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
404 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
405 _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
406 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
407
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
408 yield struct.pack(b'>QQ', filecount, bytecount)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
409 yield struct.pack(b'>H', len(requires) + 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
410 yield requires + b'\0'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
411
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
412 # This is where we'll add compression in the future.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
413 assert compression == b'UN'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
414
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
415 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
416 _(b'bundle'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
417 )
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
418 progress.update(0)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
419
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
420 for chunk in it:
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
421 progress.increment(step=len(chunk))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
422 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
423
38379
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38366
diff changeset
424 progress.complete()
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
425
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
426 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
427
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
428
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
429 def consumev1(repo, fp, filecount: int, bytecount: int) -> None:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
430 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
431
30342
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29923
diff changeset
432 This takes the output from "stream_out" and applies it to the specified
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
433 repository.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
434
30342
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29923
diff changeset
435 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29923
diff changeset
436 handled by this function.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
437 """
27859
f55a5ace8e69 with: use context manager in streamclone consumev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27850
diff changeset
438 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
439 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
440 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
441 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
442 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
443 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
444 _(b'clone'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
445 )
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
446 progress.update(0)
30995
22fbca1d11ed mercurial: switch to util.timer for all interval timings
Simon Farnsworth <simonfar@fb.com>
parents: 30342
diff changeset
447 start = util.timer()
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
448
29923
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
449 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
450 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
451 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
452 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
453 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
454 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
455 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
456 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
457 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
458 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
459 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
460 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
461 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
462
52720
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
463 total_file_count = 0
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
464 with repo.transaction(b'clone'):
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
465 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
49292
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 49222
diff changeset
466 for i in range(filecount):
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
467 # XXX doesn't support '\n' or '\r' in filenames
50951
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50706
diff changeset
468 if hasattr(fp, 'readline'):
50675
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
469 l = fp.readline()
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
470 else:
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
471 # inline clonebundles use a chunkbuffer, so no readline
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
472 # --> this should be small anyway, the first line
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
473 # only contains the size of the bundle
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
474 l_buf = []
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
475 while not (l_buf and l_buf[-1] == b'\n'):
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
476 l_buf.append(fp.read(1))
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
477 l = b''.join(l_buf)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
478 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
479 name, size = l.split(b'\0', 1)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
480 size = int(size)
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
481 except (ValueError, TypeError):
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
482 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
483 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
484 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
485 if repo.ui.debugflag:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
486 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
487 b'adding %s (%s)\n' % (name, util.bytecount(size))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
488 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
489 # for backwards compat, name was partially encoded
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
490 path = store.decodedir(name)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
491 with repo.svfs(path, b'w', backgroundclose=True) as ofp:
52720
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
492 total_file_count += 1
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
493 for chunk in util.filechunkiter(fp, limit=size):
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
494 progress.increment(step=len(chunk))
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
495 ofp.write(chunk)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
496
29923
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
497 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
498 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
499 repo.invalidate(clearfilecache=True)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
500
38379
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38366
diff changeset
501 progress.complete()
52720
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
502 _report_transferred(repo, start, total_file_count, bytecount)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
503
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
504
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
505 def readbundle1header(fp) -> tuple[int, int, Set[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
506 compression = fp.read(2)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
507 if compression != b'UN':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
508 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
509 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
510 b'only uncompressed stream clone bundles are '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
511 b'supported; got %s'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
512 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
513 % compression
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
514 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
515
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
516 filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
517 requireslen = struct.unpack(b'>H', fp.read(2))[0]
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
518 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
519
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
520 if not requires.endswith(b'\0'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
521 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
522 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
523 b'malformed stream clone bundle: '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
524 b'requirements not properly encoded'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
525 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
526 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
527
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
528 requirements = set(requires.rstrip(b'\0').split(b','))
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
529
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
530 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
531
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
532
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
533 def applybundlev1(repo, fp) -> None:
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
534 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
535
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
536 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
537 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
538 """
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
539 if len(repo):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
540 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
541 _(b'cannot apply stream clone bundle on non-empty repo')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
542 )
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
543
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
544 filecount, bytecount, requirements = readbundle1header(fp)
48359
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
545 missingreqs = requirements - repo.supported
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
546 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
547 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
548 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
549 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
550 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
551
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
552 consumev1(repo, fp, filecount, bytecount)
48707
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
553 nodemap.post_stream_cleanup(repo)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
554
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
555
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48966
diff changeset
556 class streamcloneapplier:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
557 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
558
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
559 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
560 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
561 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
562
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
563 def __init__(self, fh) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
564 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
565
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
566 def apply(self, repo) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
567 return applybundlev1(repo, self._fh)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
568
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
569
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
570 # type of file to stream
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
571 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
572 _filefull = 1 # full snapshot file
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
573
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
574 # Source of the file
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
575 _srcstore = b's' # store (svfs)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
576 _srccache = b'c' # cache (cache)
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
577
51762
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51759
diff changeset
578
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
579 # This is it's own function so extensions can override it.
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
580 def _walkstreamfullstorefiles(repo) -> list[bytes]:
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
581 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
582 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
583 if not repo.publishing():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
584 fnames.append(b'phaseroots')
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
585 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
586
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
587
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
588 def _filterfull(entry, copy, vfsmap):
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
589 """actually copy the snapshot files"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
590 src, name, ftype, data = entry
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
591 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
592 return entry
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
593 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
594
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
595
52043
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51901
diff changeset
596 class VolatileManager:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
597 """Manage temporary backups of volatile files during stream clone.
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
598
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
599 This class will keep open file handles for the volatile files, writing the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
600 smaller ones on disk if the number of open file handles grow too much.
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
601
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
602 This should be used as a Python context, the file handles and copies will
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
603 be discarded when exiting the context.
49222
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49037
diff changeset
604
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
605 The preservation can be done by calling the object on the real path
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
606 (encoded full path).
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
607
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
608 Valid filehandles for any file should be retrieved by calling `open(path)`.
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
609 """
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
610
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
611 # arbitrarily picked as "it seemed fine" and much higher than the current
52154
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52046
diff changeset
612 # usage. The Windows value of 2 is actually 1 file open at a time, due to
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52046
diff changeset
613 # the `flush_count = self.MAX_OPEN // 2` and `self.MAX_OPEN - 1` threshold
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52046
diff changeset
614 # for flushing to disk in __call__().
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52046
diff changeset
615 MAX_OPEN = 2 if pycompat.iswindows else 100
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
616
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
617 def __init__(self) -> None:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
618 self._counter = 0
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
619 self._volatile_fps = None
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
620 self._copies = None
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
621 self._dst_dir = None
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
622
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
623 def __enter__(self):
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
624 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
625 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
626 self._volatile_fps = {}
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
627 self._counter += 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
628 return self
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
629
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
630 def __exit__(self, *args, **kwars):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
631 """discard all backups"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
632 self._counter -= 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
633 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
634 for _size, fp in self._volatile_fps.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
635 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
636 self._volatile_fps = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
637 if self._copies is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
638 for tmp in self._copies.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
639 util.tryunlink(tmp)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
640 util.tryrmdir(self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
641 self._copies = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
642 self._dst_dir = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
643 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
644 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
645 assert self._dst_dir is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
646
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
647 def _init_tmp_copies(self) -> None:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
648 """prepare a temporary directory to save volatile files
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
649
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
650 This will be used as backup if we have too many files open"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
651 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
652 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
653 assert self._dst_dir is None
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
654 self._copies = {}
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
655 self._dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
656
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
657 def _flush_some_on_disk(self) -> None:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
658 """move some of the open files to tempory files on disk"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
659 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
660 self._init_tmp_copies()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
661 flush_count = self.MAX_OPEN // 2
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
662 for src, (size, fp) in sorted(self._volatile_fps.items())[:flush_count]:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
663 prefix = os.path.basename(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
664 fd, dst = pycompat.mkstemp(prefix=prefix, dir=self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
665 self._copies[src] = dst
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
666 os.close(fd)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
667 # we no longer hardlink, but on the other hand we rarely do this,
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
668 # and we do it for the smallest file only and not at all in the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
669 # common case.
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
670 with open(dst, 'wb') as bck:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
671 fp.seek(0)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
672 bck.write(fp.read())
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
673 del self._volatile_fps[src]
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
674 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
675
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
676 def _keep_one(self, src: bytes) -> int:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
677 """preserve an open file handle for a given path"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
678 # store the file quickly to ensure we close it if any error happens
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
679 _, fp = self._volatile_fps[src] = (None, open(src, 'rb'))
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
680 fp.seek(0, os.SEEK_END)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
681 size = fp.tell()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
682 self._volatile_fps[src] = (size, fp)
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
683 return size
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
684
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
685 def __call__(self, src: bytes) -> None:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
686 """preserve the volatile file at src"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
687 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
688 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
689 self._flush_some_on_disk()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
690 self._keep_one(src)
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
691
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
692 def try_keep(self, src: bytes) -> Optional[int]:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
693 """record a volatile file and returns it size
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
694
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
695 return None if the file does not exists.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
696
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
697 Used for cache file that are not lock protected.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
698 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
699 assert 0 < self._counter
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
700 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
701 self._flush_some_on_disk()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
702 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
703 return self._keep_one(src)
52665
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52391
diff changeset
704 except OSError as err:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
705 if err.errno not in (errno.ENOENT, errno.EPERM):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
706 raise
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
707 return None
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
708
52044
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52043
diff changeset
709 @contextlib.contextmanager
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52043
diff changeset
710 def open(self, src):
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
711 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
712 entry = self._volatile_fps.get(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
713 if entry is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
714 _size, fp = entry
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
715 fp.seek(0)
52044
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52043
diff changeset
716 yield fp
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
717 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
718 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
719 actual_path = src
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
720 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
721 actual_path = self._copies.get(src, src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
722 with open(actual_path, 'rb') as fp:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
723 yield fp
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
724
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
725
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
726 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
727 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
728 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
729 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
730 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
731 }
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
732 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
733 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
734 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
735
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
736 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
737
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
738
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
739 def _emit2(repo, entries):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
740 """actually emit the stream bundle"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
741 vfsmap = _makemap(repo)
47050
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
742 # we keep repo.vfs out of the on purpose, ther are too many danger there
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
743 # (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
744 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
745 # this assert is duplicated (from _makemap) as author might think this is
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
746 # fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
747 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
748 raise error.ProgrammingError(
52714
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52713
diff changeset
749 'repo.vfs must not be added to vfsmap for security reasons'
47050
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
750 )
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
751
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
752 # translate the vfs one
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
753 entries = [(vfs_key, vfsmap[vfs_key], e) for (vfs_key, e) in entries]
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
754 _test_sync_point_walk_1_2(repo)
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
755
50670
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
756 max_linkrev = len(repo)
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
757 file_count = totalfilesize = 0
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
758 with VolatileManager() as volatiles:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
759 # make sure we preserve volatile files
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
760 with util.nogc():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
761 # record the expected size of every file
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
762 for k, vfs, e in entries:
52389
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52388
diff changeset
763 e.preserve_volatiles(vfs, volatiles)
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
764 for f in e.files():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
765 file_count += 1
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
766 totalfilesize += f.file_size(vfs)
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
767
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
768 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
769 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
770 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
771 progress.update(0)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
772 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
773 # the first yield release the lock on the repository
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
774 yield file_count, totalfilesize
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
775 totalbytecount = 0
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
776
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
777 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
778 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
779 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
780 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
781 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
782 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
783 preserve_file_count=True,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
784 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
785 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
786 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
787 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
788 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
789 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
790 bytecount = 0
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
791 for chunk in stream:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
792 bytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
793 totalbytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
794 progress.update(totalbytecount)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
795 yield chunk
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
796 if bytecount != size:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
797 # Would most likely be caused by a race due to `hg
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
798 # strip` or a revlog split
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
799 msg = _(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
800 b'clone could only read %d bytes from %s, but '
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
801 b'expected %d bytes'
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
802 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
803 raise error.Abort(msg % (bytecount, name, size))
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
804
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
805
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
806 def _emit3(repo, entries) -> Iterator[bytes | None]:
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
807 """actually emit the stream bundle (v3)"""
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
808 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
809 # we keep repo.vfs out of the map on purpose, ther are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
810 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
811 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
812 # this assert is duplicated (from _makemap) as authors might think this is
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
813 # fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
814 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
815 raise error.ProgrammingError(
52714
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52713
diff changeset
816 'repo.vfs must not be added to vfsmap for security reasons'
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
817 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
818
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
819 # translate the vfs once
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
820 # we only turn this into a list for the `_test_sync`, this is not ideal
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
821 base_entries = list(entries)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
822 _test_sync_point_walk_1_2(repo)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
823 entries = []
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
824 with VolatileManager() as volatiles:
52043
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51901
diff changeset
825 # make sure we preserve volatile files
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
826 for vfs_key, e in base_entries:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
827 vfs = vfsmap[vfs_key]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
828 any_files = True
51590
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
829 if e.maybe_volatile:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
830 any_files = False
52389
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52388
diff changeset
831 e.preserve_volatiles(vfs, volatiles)
51590
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
832 for f in e.files():
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
833 if f.is_volatile:
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
834 # record the expected size under lock
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
835 f.file_size(vfs)
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
836 any_files = True
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
837 if any_files:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
838 entries.append((vfs_key, vfsmap[vfs_key], e))
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
839
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
840 total_entry_count = len(entries)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
841
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
842 max_linkrev = len(repo)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
843 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
844 _(b'bundle'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
845 total=total_entry_count,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
846 unit=_(b'entry'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
847 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
848 progress.update(0)
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
849 # the first yield release the lock on the repository
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
850 yield None
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
851 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
852 yield util.uvarintencode(total_entry_count)
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
853
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
854 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
855 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
856 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
857 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
858 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
859 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
860 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
861 yield util.uvarintencode(len(entry_streams))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
862 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
863 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
864 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
865 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
866 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
867 yield from stream
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
868 progress.increment()
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
869
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
870
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
871 def _test_sync_point_walk_1_2(repo):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
872 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
873
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
874 Triggered after gather entry, but before starting to process/preserve them
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
875 under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
876
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
877 (on v1 is triggered before the actual walk start)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
878 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
879
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
880
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
881 def _test_sync_point_walk_3(repo):
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
882 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
883
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
884 Triggered right before releasing the lock, but after computing what need
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
885 needed to compute under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
886 """
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
887
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
888
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
889 def _test_sync_point_walk_4(repo):
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
890 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
891
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
892 Triggered right after releasing the lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
893 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
894
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
895
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
896 # not really a StoreEntry, but close enough
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
897 class CacheEntry(store.SimpleStoreEntry):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
898 """Represent an entry for Cache files
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
899
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
900 It has special logic to preserve cache file early and accept optional
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
901 presence.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
902
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
903
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
904 (Yes... this is not really a StoreEntry, but close enough. We could have a
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
905 BaseEntry base class, bbut the store one would be identical)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
906 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
907
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
908 def __init__(self, entry_path) -> None:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
909 super().__init__(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
910 entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
911 # we will directly deal with that in `setup_cache_file`
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
912 is_volatile=True,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
913 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
914
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
915 def preserve_volatiles(self, vfs, volatiles) -> None:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
916 self._file_size = volatiles.try_keep(vfs.join(self._entry_path))
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
917 if self._file_size is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
918 self._files = []
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
919 else:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
920 assert self._is_volatile
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
921 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
922 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
923 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
924 file_size=self._file_size,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
925 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
926 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
927 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
928
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
929 def files(self) -> list[store.StoreFile]:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
930 if self._files is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
931 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
932 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
933 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
934 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
935 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
936 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
937 return super().files()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
938
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
939
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
940 class CacheFile(store.StoreFile):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
941 # inform the "copy/hardlink" version that this file might be missing
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
942 # without consequences.
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
943 optional: bool = True
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
944
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
945
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
946 def _entries_walk(repo, includes, excludes, includeobsmarkers: bool):
50549
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
947 """emit a seris of files information useful to clone a repo
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
948
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
949 return (vfs-key, entry) iterator
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
950
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
951 Where `entry` is StoreEntry. (used even for cache entries)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
952 """
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
953 assert repo._currentlock(repo._lockref) is not None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
954
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
955 matcher = None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
956 if includes or excludes:
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
957 matcher = narrowspec.match(repo.root, includes, excludes)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
958
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
959 phase = not repo.publishing()
51588
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
960 # Python is getting crazy at all the small container we creates, disabling
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
961 # the gc while we do so helps performance a lot.
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
962 with util.nogc():
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
963 entries = _walkstreamfiles(
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
964 repo,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
965 matcher,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
966 phase=phase,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
967 obsolescence=includeobsmarkers,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
968 )
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
969 for entry in entries:
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
970 yield (_srcstore, entry)
50549
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
971
51588
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
972 for name in cacheutil.cachetocopy(repo):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
973 if repo.cachevfs.exists(name):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
974 # not really a StoreEntry, but close enough
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
975 yield (_srccache, CacheEntry(entry_path=name))
50550
43ed1f12b00a stream-clone: yield cache entry in `_entries_walk` too
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50549
diff changeset
976
50549
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
977
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
978 def generatev2(repo, includes, excludes, includeobsmarkers: bool):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
979 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
980
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
981 the data stream consists the following entries:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
982 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
983 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
984 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
985 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
986 5) N bytes containing the file data
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
987
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
988 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
989 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
990
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
991 with repo.lock():
47450
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
992 repo.ui.debug(b'scanning\n')
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
993
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
994 entries = _entries_walk(
47450
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
995 repo,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
996 includes=includes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
997 excludes=excludes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
998 includeobsmarkers=includeobsmarkers,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
999 )
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
1000 chunks = _emit2(repo, entries)
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
1001 first = next(chunks)
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
1002 file_count, total_file_size = first
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
1003 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
1004 _test_sync_point_walk_4(repo)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1005
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
1006 return file_count, total_file_size, chunks
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1007
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1008
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1009 def generatev3(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1010 repo, includes, excludes, includeobsmarkers: bool
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1011 ) -> Iterator[bytes | None]:
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1012 """Emit content for version 3 of a streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1013
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1014 the data stream consists the following:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1015 1) A varint E containing the number of entries (can be 0), then E entries follow
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1016 2) For each entry:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1017 2.1) The number of files in this entry (can be 0, but typically 1 or 2)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1018 2.2) For each file:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1019 2.2.1) A char representing the file destination (eg: store or cache)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1020 2.2.2) A varint N containing the length of the filename
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1021 2.2.3) A varint M containing the length of file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1022 2.2.4) N bytes containing the filename (the internal, store-agnostic form)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1023 2.2.5) M bytes containing the file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1024
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1025 Returns the data iterator.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1026
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1027 XXX This format is experimental and subject to change. Here is a
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1028 XXX non-exhaustive list of things this format could do or change:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1029
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1030 - making it easier to write files in parallel
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1031 - holding the lock for a shorter time
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1032 - improving progress information
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1033 - ways to adjust the number of expected entries/files ?
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1034 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1035
51589
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51588
diff changeset
1036 # Python is getting crazy at all the small container we creates while
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51588
diff changeset
1037 # considering the files to preserve, disabling the gc while we do so helps
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51588
diff changeset
1038 # performance a lot.
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51588
diff changeset
1039 with repo.lock(), util.nogc():
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1040 repo.ui.debug(b'scanning\n')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1041
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1042 entries = _entries_walk(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1043 repo,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1044 includes=includes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1045 excludes=excludes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1046 includeobsmarkers=includeobsmarkers,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1047 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1048 chunks = _emit3(repo, list(entries))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1049 first = next(chunks)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1050 assert first is None
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
1051 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
1052 _test_sync_point_walk_4(repo)
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1053
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1054 return chunks
50558
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50557
diff changeset
1055
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50557
diff changeset
1056
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1057 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1058 def nested(*ctxs):
39773
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1059 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1060 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1061 with this:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1062 if rest:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1063 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1064 yield
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1065 else:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1066 yield
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1067
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1068
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1069 class V2Report:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1070 """a small class to track the data we saw within the stream"""
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1071
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1072 def __init__(self):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1073 self.byte_count = 0
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1074
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1075
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1076 def consumev2(repo, fp, filecount: int, filesize: int) -> None:
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1077 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1078
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1079 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1080 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1081 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1082 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1083 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1084 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1085 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1086 )
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1087 progress = repo.ui.makeprogress(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1088 _(b'clone'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1089 total=filesize,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1090 unit=_(b'bytes'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1091 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1092 start = util.timer()
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1093 report = V2Report()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1094
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1095 vfsmap = _makemap(repo)
47050
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1096 # we keep repo.vfs out of the on purpose, ther are too many danger
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1097 # there (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1098 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1099 # this assert is duplicated (from _makemap) as author might think this
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1100 # is fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1101 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1102 raise error.ProgrammingError(
52714
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52713
diff changeset
1103 'repo.vfs must not be added to vfsmap for security reasons'
47050
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1104 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1105
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1106 threaded = repo.ui.configbool(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1107 b"worker", b"parallel-stream-bundle-processing"
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1108 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1109 num_writer = repo.ui.configint(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1110 b"worker",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1111 b"parallel-stream-bundle-processing.num-writer",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1112 )
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1113 with repo.transaction(b'clone'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1114 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1115 with nested(*ctxs):
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1116 if threaded:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1117 fc = _ThreadSafeFileChunker
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1118 else:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1119 fc = _FileChunker
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1120 files = _v2_parse_files(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1121 repo,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1122 fp,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1123 filecount,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1124 progress,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1125 report,
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1126 file_chunker=fc,
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1127 )
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1128 if not threaded:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1129 _write_files(vfsmap, files)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1130 else:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1131 info_queue = _FileInfoQueue(files)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1132
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1133 workers = []
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1134 try:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1135 for __ in range(num_writer):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1136 w = threading.Thread(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1137 target=_write_files,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1138 args=(vfsmap, info_queue),
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1139 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1140 workers.append(w)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1141 w.start()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1142 info_queue.fill()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1143 finally:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1144 # shut down all the workers
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1145 info_queue.close(len(workers))
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1146 for w in workers:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1147 w.join()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1148
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1149 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1150 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1151 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1152
38379
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38366
diff changeset
1153 progress.complete()
52721
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52720
diff changeset
1154 # acknowledge the end of the bundle2 part, this help aligning
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52720
diff changeset
1155 # sequential and parallel behavior.
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52720
diff changeset
1156 remains = fp.read(1)
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52720
diff changeset
1157 assert not remains
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1158 _report_transferred(repo, start, filecount, report.byte_count)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1159
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1160
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1161 # iterator of chunk of bytes that constitute a file content.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1162 FileChunksT = Iterator[bytes]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1163 # Contains the information necessary to write stream file on disk
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1164 FileInfoT = Tuple[
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1165 bytes, # vfs key
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1166 bytes, # file name (non-vfs-encoded)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1167 FileChunksT, # content
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1168 ]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1169
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1170
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1171 class _FileInfoQueue:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1172 """A thread-safe queue to passer parsed file information to the writers"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1173
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1174 def __init__(self, info: Iterable[FileInfoT]):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1175 self._info = info
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1176 self._q = queue.Queue()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1177
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1178 def fill(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1179 """iterate over the parsed information to file the queue
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1180
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1181 This is meant to be call from the thread parsing the stream information.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1182 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1183 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1184 for i in self._info:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1185 q.put(i)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1186
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1187 def close(self, number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1188 """signal all the workers that we no longer have any file info coming
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1189
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1190 Called from the thread parsing the stream information (and/or the main
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1191 thread if different).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1192 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1193 for __ in range(number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1194 self._q.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1195
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1196 def __iter__(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1197 """iterate over the available file info
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1198
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1199 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1200 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1201 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1202 while (i := q.get()) is not None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1203 yield i
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1204
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1205
52923
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1206 class _FileChunker:
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1207 """yield the chunk that constitute a file
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1208
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1209 This class exists as the counterpart of the threaded version and
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1210 would not be very useful on its own.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1211 """
52923
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1212
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1213 def __init__(
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1214 self,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1215 fp: bundle2mod.unbundlepart,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1216 data_len: int,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1217 progress: scmutil.progress,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1218 report: V2Report,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1219 ):
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1220 self.report = report
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1221 self.progress = progress
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1222 self._chunks = util.filechunkiter(fp, limit=data_len)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1223
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1224 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1225 """Do nothing in non-threading context"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1226
52923
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1227 def __iter__(self) -> FileChunksT:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1228 for chunk in self._chunks:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1229 self.report.byte_count += len(chunk)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1230 self.progress.increment(step=len(chunk))
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1231 yield chunk
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1232
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1233
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1234 class _ThreadSafeFileChunker(_FileChunker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1235 """yield the chunk that constitute a file
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1236
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1237 Make sure you call the "fill" function in the main thread to read the
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1238 right data at the right time.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1239 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1240
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1241 def __init__(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1242 self,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1243 fp: bundle2mod.unbundlepart,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1244 data_len: int,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1245 progress: scmutil.progress,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1246 report: V2Report,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1247 ):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1248 super().__init__(fp, data_len, progress, report)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1249 self._queue = queue.Queue()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1250
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1251 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1252 """fill the file chunker queue with data read from the stream
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1253
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1254 This is meant to be called from the thread parsing information (and
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1255 consuming the stream data).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1256 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1257 try:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1258 for chunk in super().__iter__():
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1259 self._queue.put(chunk)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1260 finally:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1261 self._queue.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1262
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1263 def __iter__(self) -> FileChunksT:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1264 """Iterate over all the file chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1265
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1266 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1267 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1268 while (chunk := self._queue.get()) is not None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1269 yield chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1270
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1271
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1272 def _v2_parse_files(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1273 repo,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1274 fp: bundle2mod.unbundlepart,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1275 file_count: int,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1276 progress: scmutil.progress,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1277 report: V2Report,
52923
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1278 file_chunker: Type[_FileChunker] = _FileChunker,
52921
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1279 ) -> Iterator[FileInfoT]:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1280 """do the "stream-parsing" part of stream v2
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1281
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1282 The parsed information are yield result for consumption by the "writer"
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1283 """
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1284 progress.update(0)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1285 for i in range(file_count):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1286 src = util.readexactly(fp, 1)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1287 namelen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1288 datalen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1289
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1290 name = util.readexactly(fp, namelen)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1291
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1292 if repo.ui.debugflag:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1293 repo.ui.debug(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1294 b'adding [%s] %s (%s)\n' % (src, name, util.bytecount(datalen))
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52721
diff changeset
1295 )
52923
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1296 chunks = file_chunker(fp, datalen, progress, report)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1297 yield (src, name, iter(chunks))
52924
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1298 # make sure we read all the chunk before moving to the next file
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1299 chunks.fill()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1300
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1301
52922
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1302 def _write_files(vfsmap, info: Iterable[FileInfoT]):
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1303 """write files from parsed data"""
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1304 for src, name, data in info:
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1305 vfs = vfsmap[src]
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1306 with vfs(name, b'w') as ofp:
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1307 for chunk in data:
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1308 ofp.write(chunk)
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1309
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1310
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1311 def consumev3(repo, fp) -> None:
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1312 """Apply the contents from a version 3 streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1313
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1314 Data is read from an object that only needs to provide a ``read(size)``
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1315 method.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1316 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1317 with repo.lock():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1318 start = util.timer()
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1319
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1320 entrycount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1321 repo.ui.status(_(b'%d entries to transfer\n') % (entrycount))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1322
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1323 progress = repo.ui.makeprogress(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1324 _(b'clone'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1325 total=entrycount,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1326 unit=_(b'entries'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1327 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1328 progress.update(0)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1329 bytes_transferred = 0
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1330
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1331 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1332 # we keep repo.vfs out of the on purpose, there are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1333 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1334 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1335 # this assert is duplicated (from _makemap) as authors might think this
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1336 # is fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1337 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1338 raise error.ProgrammingError(
52714
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52713
diff changeset
1339 'repo.vfs must not be added to vfsmap for security reasons'
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1340 )
52720
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
1341 total_file_count = 0
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1342 with repo.transaction(b'clone'):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1343 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1344 with nested(*ctxs):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1345 for i in range(entrycount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1346 filecount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1347 if filecount == 0:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1348 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1349 repo.ui.debug(b'entry with no files [%d]\n' % (i))
52720
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
1350 total_file_count += filecount
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1351 for i in range(filecount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1352 src = util.readexactly(fp, 1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1353 vfs = vfsmap[src]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1354 namelen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1355 datalen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1356
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1357 name = util.readexactly(fp, namelen)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1358
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1359 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1360 msg = b'adding [%s] %s (%s)\n'
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1361 msg %= (src, name, util.bytecount(datalen))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1362 repo.ui.debug(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1363 bytes_transferred += datalen
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1364
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1365 with vfs(name, b'w') as ofp:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1366 for chunk in util.filechunkiter(fp, limit=datalen):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1367 ofp.write(chunk)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1368 progress.increment(step=1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1369
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1370 # force @filecache properties to be reloaded from
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1371 # streamclone-ed file at next access
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1372 repo.invalidate(clearfilecache=True)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1373
52719
6feb3b3029b5 stream: consistently close progress before reporting time
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52718
diff changeset
1374 progress.complete()
52720
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52719
diff changeset
1375 _report_transferred(repo, start, total_file_count, bytes_transferred)
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1376
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1377
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1378 def applybundlev2(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1379 repo, fp, filecount: int, filesize: int, requirements: Iterable[bytes]
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1380 ) -> None:
39716
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1381 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1382
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1383 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1384 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1385 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
1386 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1387 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1388 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1389
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1390 consumev2(repo, fp, filecount, filesize)
35804
2d3e486d09d0 streamclone: move requirement update into consumev2
Boris Feld <boris.feld@octobus.net>
parents: 35803
diff changeset
1391
48618
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
1392 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
1393 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
1394 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1395 )
39716
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1396 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1397 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1398 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
1399 scmutil.writereporequirements(repo)
48707
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
1400 nodemap.post_stream_cleanup(repo)
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1401
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1402
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1403 def applybundlev3(repo, fp, requirements: Iterable[bytes]) -> None:
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1404 from . import localrepo
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1405
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1406 missingreqs = [r for r in requirements if r not in repo.supported]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1407 if missingreqs:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1408 msg = _(b'unable to apply stream clone: unsupported format: %s')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1409 msg %= b', '.join(sorted(missingreqs))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1410 raise error.Abort(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1411
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1412 consumev3(repo, fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1413
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1414 repo.requirements = new_stream_clone_requirements(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1415 repo.requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1416 requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1417 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1418 repo.svfs.options = localrepo.resolvestorevfsoptions(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1419 repo.ui, repo.requirements, repo.features
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1420 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1421 scmutil.writereporequirements(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1422 nodemap.post_stream_cleanup(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1423
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1424
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1425 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress) -> bool:
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1426 hardlink = [True]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1427
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1428 def copy_used():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1429 hardlink[0] = False
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1430 progress.topic = _(b'copying')
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1431
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1432 for k, path, optional in entries:
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1433 src_vfs = src_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1434 dst_vfs = dst_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1435 src_path = src_vfs.join(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1436 dst_path = dst_vfs.join(path)
47863
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1437 # We cannot use dirname and makedirs of dst_vfs here because the store
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1438 # encoding confuses them. See issue 6581 for details.
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1439 dirname = os.path.dirname(dst_path)
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1440 if not os.path.exists(dirname):
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1441 util.makedirs(dirname)
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1442 dst_vfs.register_file(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1443 # XXX we could use the #nb_bytes argument.
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1444 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1445 util.copyfile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1446 src_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1447 dst_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1448 hardlink=hardlink[0],
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1449 no_hardlink_cb=copy_used,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1450 check_fs_hardlink=False,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1451 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1452 except FileNotFoundError:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1453 if not optional:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1454 raise
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1455 progress.increment()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1456 return hardlink[0]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1457
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1458
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1459 def local_copy(src_repo, dest_repo) -> None:
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1460 """copy all content from one local repository to another
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1461
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1462 This is useful for local clone"""
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1463 src_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1464 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1465 for r in src_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1466 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1467 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1468 dest_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1469 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1470 for r in dest_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1471 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1472 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1473 assert src_store_requirements == dest_store_requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1474
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1475 with dest_repo.lock():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1476 with src_repo.lock():
47454
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1477 # bookmark is not integrated to the streaming as it might use the
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1478 # `repo.vfs` and they are too many sentitive data accessible
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1479 # through `repo.vfs` to expose it to streaming clone.
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1480 src_book_vfs = bookmarks.bookmarksvfs(src_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1481 srcbookmarks = src_book_vfs.join(b'bookmarks')
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1482 bm_count = 0
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1483 if os.path.exists(srcbookmarks):
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1484 bm_count = 1
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1485
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1486 entries = _entries_walk(
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1487 src_repo,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1488 includes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1489 excludes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1490 includeobsmarkers=True,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1491 )
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1492 entries = list(entries)
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1493 src_vfs_map = _makemap(src_repo)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1494 dest_vfs_map = _makemap(dest_repo)
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1495 total_files = sum(len(e[1].files()) for e in entries) + bm_count
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1496 progress = src_repo.ui.makeprogress(
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1497 topic=_(b'linking'),
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1498 total=total_files,
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1499 unit=_(b'files'),
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1500 )
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1501 # copy files
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1502 #
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1503 # We could copy the full file while the source repository is locked
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1504 # and the other one without the lock. However, in the linking case,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1505 # this would also requires checks that nobody is appending any data
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1506 # to the files while we do the clone, so this is not done yet. We
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1507 # could do this blindly when copying files.
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1508 files = [
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1509 (vfs_key, f.unencoded_path, f.optional)
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1510 for vfs_key, e in entries
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1511 for f in e.files()
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1512 ]
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1513 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1514
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1515 # copy bookmarks over
47454
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1516 if bm_count:
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1517 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1518 dstbookmarks = dst_book_vfs.join(b'bookmarks')
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1519 util.copyfile(srcbookmarks, dstbookmarks)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1520 progress.complete()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1521 if hardlink:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1522 msg = b'linked %d files\n'
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1523 else:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1524 msg = b'copied %d files\n'
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1525 src_repo.ui.debug(msg % total_files)
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1526
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1527 with dest_repo.transaction(b"localclone") as tr:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1528 dest_repo.store.write(tr)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1529
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1530 # clean up transaction file as they do not make sense
50288
d89eecf9605e undo-files: no longer pass the `repo` to `cleanup_undo_files`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50286
diff changeset
1531 transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)