Mercurial > public > mercurial-scm > hg

annotate mercurial/streamclone.py @ 52914:d5ae681834e8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
stream-clone-v2: also use a thread to read the streamed data We could do useful thing while waiting for data, so we introduce another thread dedicated to reading the bundle data and pass them to the thread parsing the bundle. This has a small impact on test output because the bundle part is consumed at a different time. However this will be smoothed out very soon so lets ignore that for now. As for the previous patch, the speed up in not there yet. We need to adjust various part of the implementation to see some benefit. However, this put the generic architecture in place so that we can focus on performance later. Below are some benchmark result to highlight the slowdown. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog no-thread: 7.244015 ~~~~~ write-thread: 9.128669 (+26.02%, +1.88) read-thread: 9.387256 (+29.59%, +2.14) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog no-thread: 0.249693 ~~~~~ write-thread: 0.275081 (+10.17%, +0.03) read-thread: 0.292601 (+17.18%, +0.04) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog no-thread: 13.136674 ~~~~~ write-thread: 16.374306 (+24.65%, +3.24) read-thread: 17.008865 (+29.48%, +3.87) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog no-thread: 5.317709 ~~~~~ write-thread: 6.783031 (+27.56%, +1.47) read-thread: 7.107141 (+33.65%, +1.79) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog no-thread: 5.398368 ~~~~~ write-thread: 6.737864 (+24.81%, +1.34) read-thread: 7.163505 (+32.70%, +1.77) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog no-thread: 3.acbb55 ~~~~~ write-thread: 3.758324 (+22.48%, +0.69) read-thread: 3.907693 (+27.34%, +0.84) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-thread: 51.934795 ~~~~~ write-thread: 66.561340 (+28.16%, +14.63) read-thread: 66.902619 (+28.82%, +14.97) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog no-thread: 52.253858 ~~~~~ write-thread: 66.955037 (+28.13%, +14.70) read-thread: 66.397609 (+27.07%, +14.14) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no no-thread: 130.584329 ~~~~~ write-thread: 138.770454 (+6.27%, +8.19) read-thread: 145.137477 (+11.14%, +14.55)
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Mon, 20 Jan 2025 17:05:22 +0100
parents 58baa86c7a02
children 9abf173a958b
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
51859
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
8 from __future__ import annotations
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
10 import contextlib
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
11 import errno
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
12 import os
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
13 import queue
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
14 import struct
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
15 import threading
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
16
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
17 from typing import (
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
18 Iterable,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
19 Iterator,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
20 Optional,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
21 Set,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
22 Tuple,
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
23 Type,
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
24 )
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
25
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
26 from .i18n import _
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
27 from .interfaces import repository
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
28 from . import (
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
29 bookmarks,
50520
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
30 bundle2 as bundle2mod,
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
31 cacheutil,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
32 error,
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
33 narrowspec,
32744
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
34 phases,
38164
aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path
Yuya Nishihara <yuya@tcha.org>
parents: 37638
diff changeset
35 pycompat,
46626
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
36 requirements as requirementsmod,
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
37 scmutil,
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
38 store,
50286
3d0b5760851c undo-files: move the undo cleanup code in the transaction module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50279
diff changeset
39 transaction,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
40 util,
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
41 )
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
42 from .revlogutils import (
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
43 nodemap,
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
44 )
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
45
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
46
52913
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
47 # Number arbitrarily picked, feel free to change them (but the LOW one)
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
48 #
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
49 # update the configuration documentation if you touch this.
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
50 DEFAULT_NUM_WRITER = {
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
51 scmutil.RESOURCE_LOW: 1,
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
52 scmutil.RESOURCE_MEDIUM: 2,
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
53 scmutil.RESOURCE_HIGH: 4,
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
54 }
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
55
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
56
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
57 def new_stream_clone_requirements(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
58 default_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
59 streamed_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
60 ) -> Set[bytes]:
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
61 """determine the final set of requirement for a new stream clone
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
62
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
63 this method combine the "default" requirements that a new repository would
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
64 use with the constaint we get from the stream clone content. We keep local
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
65 configuration choice when possible.
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
66 """
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
67 requirements = set(default_requirements)
48601
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
68 requirements -= requirementsmod.STREAM_FIXED_REQUIREMENTS
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
69 requirements.update(streamed_requirements)
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
70 return requirements
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
71
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
72
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
73 def streamed_requirements(repo) -> Set[bytes]:
48597
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
74 """the set of requirement the new clone will have to support
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
75
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
76 This is used for advertising the stream options and to generate the actual
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
77 stream content."""
48601
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
78 requiredformats = (
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
79 repo.requirements & requirementsmod.STREAM_FIXED_REQUIREMENTS
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
80 )
48597
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
81 return requiredformats
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
82
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
83
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
84 def canperformstreamclone(pullop, bundle2: bool = False):
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
85 """Whether it is possible to perform a streaming clone as part of pull.
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
86
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
87 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
88 bundle2 and only through bundle2.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
89
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
90 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
91 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
92 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
93 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
94 """
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
95 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
96 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
97
50519
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
98 # should we consider streaming clone at all ?
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
99 streamrequested = pullop.streamclonerequested
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
100 # If we don't have a preference, let the server decide for us. This
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
101 # likely only comes into play in LANs.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
102 if streamrequested is None:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
103 # The server can advertise whether to prefer streaming clone.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
104 streamrequested = remote.capable(b'stream-preferred')
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
105 if not streamrequested:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
106 return False, None
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
107
50517
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
108 # Streaming clone only works on an empty destination repository
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
109 if len(repo):
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
110 return False, None
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
111
50518
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
112 # Streaming clone only works if all data is being requested.
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
113 if pullop.heads:
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
114 return False, None
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
115
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
116 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
117 if pullop.canusebundle2:
50520
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
118 local_caps = bundle2mod.getrepocaps(repo, role=b'client')
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
119 local_supported = set(local_caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
120 remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
121 bundle2supported = bool(local_supported & remote_supported)
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
122 # else
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
123 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
124 # the versions we support. Fall back and possibly allow legacy.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
125
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
126 # Ensures legacy code path uses available bundle2.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
127 if bundle2supported and not bundle2:
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
128 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
129 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
130 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
131 return False, None
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
132
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
133 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
134 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
135 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
136 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
137 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
138 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
139 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
140 requirements = set()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
141 if remote.capable(b'stream'):
46626
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
142 requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
143 else:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
144 streamreqs = remote.capable(b'streamreqs')
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
145 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
146 if not streamreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
147 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
148 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
149 b'warning: stream clone requested but server has them '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
150 b'disabled\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
151 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
152 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
153 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
154
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
155 streamreqs = set(streamreqs.split(b','))
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
156 # Server requires something we don't support. Bail.
48670
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
157 missingreqs = streamreqs - repo.supported
32259
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30975
diff changeset
158 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30975
diff changeset
159 pullop.repo.ui.warn(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
160 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
161 b'warning: stream clone requested but client is missing '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
162 b'requirements: %s\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
163 )
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
164 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
165 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
166 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
167 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
168 b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
169 b'for more information)\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
170 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
171 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
172 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
173 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
174
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
175 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
176
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
177
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
178 def maybeperformlegacystreamclone(pullop) -> None:
26462
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
179 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
180
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
181 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
182 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
183
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
184 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
185 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
186 """
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
187 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
188
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
189 supported, requirements = canperformstreamclone(pullop)
26458
362793295640 streamclone: refactor maybeperformstreamclone to take a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26447
diff changeset
190
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
191 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
192 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
193
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
194 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
195 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
196
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
197 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
198 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
199 rbranchmap = None
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
200 if remote.capable(b'branchmap'):
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
201 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
202 rbranchmap = e.callcommand(b'branchmap', {}).result()
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
203
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
204 repo.ui.status(_(b'streaming all changes\n'))
26470
4b5647d9ee13 streamclone: move "streaming all changes" message location
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26469
diff changeset
205
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
206 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
207 fp = e.callcommand(b'stream_out', {}).result()
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
208
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
209 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
210 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
211 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
212 # doesn't matter.
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
213 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
214 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
215 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
216 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
217 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
218 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
219 )
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
220 if resp == 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
221 raise error.Abort(_(b'operation forbidden by server'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
222 elif resp == 2:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
223 raise error.Abort(_(b'locking the remote repository failed'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
224 elif resp != 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
225 raise error.Abort(_(b'the server sent an unknown error code'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
226
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
227 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
228 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
229 filecount, bytecount = map(int, l.split(b' ', 1))
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
230 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
231 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
232 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
233 )
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
234
27850
49cfddbf54ba with: use context manager in maybeperformlegacystreamclone
Bryan O'Sullivan <bryano@fb.com>
parents: 27845
diff changeset
235 with repo.lock():
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
236 consumev1(repo, fp, filecount, bytecount)
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
237 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
238 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
239 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
240 )
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
241 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
242 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
243 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
244 scmutil.writereporequirements(repo)
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
245 nodemap.post_stream_cleanup(repo)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
246
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
247 if rbranchmap:
41615
328ca3b9e545 branchmap: encapsulate cache updating in the map itself
Martijn Pieters <mj@octobus.net>
parents: 40494
diff changeset
248 repo._branchcaches.replace(repo, rbranchmap)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
249
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
250 repo.invalidate()
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
251
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
252
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
253 def allowservergeneration(repo) -> bool:
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
254 """Whether streaming clones are allowed from the server."""
40028
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
255 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
256 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
257
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
258 if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
32744
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
259 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
260
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
261 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
262 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
263 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
264 if secret:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
265 return repo.ui.configbool(b'server', b'uncompressedallowsecret')
32744
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
266
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
267 return True
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
268
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
269
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
270 # This is it's own function so extensions can override it.
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
271 def _walkstreamfiles(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
272 repo, matcher=None, phase: bool = False, obsolescence: bool = False
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
273 ):
50511
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50509
diff changeset
274 return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
275
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
276
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
277 def _report_transferred(
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
278 repo, start_time: float, file_count: int, byte_count: int
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
279 ):
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
280 """common utility to report time it took to apply the stream bundle"""
52700
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
281 elapsed = util.timer() - start_time
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
282 if elapsed <= 0:
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
283 elapsed = 0.001
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
284 m = _(b'stream-cloned %d files / %s in %.1f seconds (%s/sec)\n')
52700
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
285 m %= (
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
286 file_count,
52700
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
287 util.bytecount(byte_count),
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
288 elapsed,
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
289 util.bytecount(byte_count / elapsed),
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
290 )
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
291 repo.ui.status(m)
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
292
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
293
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
294 def generatev1(repo) -> tuple[int, int, Iterator[bytes]]:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
295 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
296
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
297 This returns a 3-tuple of (file count, byte size, data iterator).
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
298
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
299 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
300 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
301 delimited by a null byte.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
302
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
303 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
304 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
305
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
306 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
307 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
308 for adding it.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
309
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
310 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
311 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
312 """
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
313 entries = []
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
314 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
315 # Get consistent snapshot of repo, lock during scan.
27845
7417e1c10253 with: use context manager in streamclone generatev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27794
diff changeset
316 with repo.lock():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
317 repo.ui.debug(b'scanning\n')
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
318 _test_sync_point_walk_1_2(repo)
50468
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50288
diff changeset
319 for entry in _walkstreamfiles(repo):
50469
9fdc28e21b68 store: introduce a EntryFile object to actually access file info
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50468
diff changeset
320 for f in entry.files():
50474
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
321 file_size = f.file_size(repo.store.vfs)
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
322 if file_size:
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
323 entries.append((f.unencoded_path, file_size))
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
324 total_bytes += file_size
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
325 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
326 _test_sync_point_walk_4(repo)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
327
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
328 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
329 b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
330 )
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
331
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
332 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
333 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
334
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
335 def emitrevlogdata() -> Iterator[bytes]:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
336 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
337 if debugflag:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
338 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
339 # partially encode name over the wire for backwards compat
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
340 yield b'%s\0%d\n' % (store.encodedir(name), size)
33411
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
341 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
342 # trusted by the local repo) and expensive
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
343 with svfs(name, b'rb', auditpath=False) as fp:
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
344 if size <= 65536:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
345 yield fp.read(size)
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
346 else:
52644
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52640
diff changeset
347 yield from util.filechunkiter(fp, limit=size)
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
348
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
349 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
350
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
351
52691
279e217d6041 typing: lock in the new type annotations detected with the pyupgrade changes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52644
diff changeset
352 def generatev1wireproto(repo) -> Iterator[bytes]:
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
353 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
354
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
355 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
356 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
357 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
358
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
359 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
360 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
361 a permissions error for the server process).
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
362 """
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
363 if not allowservergeneration(repo):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
364 yield b'1\n'
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
365 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
366
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
367 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
368 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
369 except error.LockError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
370 yield b'2\n'
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
371 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
372
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
373 # Indicates successful response.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
374 yield b'0\n'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
375 yield b'%d %d\n' % (filecount, bytecount)
52644
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52640
diff changeset
376 yield from it
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
377
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
378
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
379 def generatebundlev1(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
380 repo, compression: bytes = b'UN'
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
381 ) -> tuple[Set[bytes], Iterator[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
382 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
383
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
384 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
385 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
386
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
387 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
388 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
389
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
390 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
391 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
392
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
393 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
394 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
395 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
396 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
397
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
398 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
399 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
400
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
401 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
402 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
403 if compression != b'UN':
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
404 raise ValueError('we do not support the compression argument yet')
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
405
48597
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
406 requirements = streamed_requirements(repo)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
407 requires = b','.join(sorted(requirements))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
408
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
409 def gen() -> Iterator[bytes]:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
410 yield b'HGS1'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
411 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
412
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
413 filecount, bytecount, it = generatev1(repo)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
414 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
415 _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
416 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
417
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
418 yield struct.pack(b'>QQ', filecount, bytecount)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
419 yield struct.pack(b'>H', len(requires) + 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
420 yield requires + b'\0'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
421
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
422 # This is where we'll add compression in the future.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
423 assert compression == b'UN'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
424
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
425 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
426 _(b'bundle'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
427 )
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
428 progress.update(0)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
429
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
430 for chunk in it:
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
431 progress.increment(step=len(chunk))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
432 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
433
38373
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38360
diff changeset
434 progress.complete()
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
435
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
436 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
437
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
438
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
439 def consumev1(repo, fp, filecount: int, bytecount: int) -> None:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
440 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
441
30332
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29919
diff changeset
442 This takes the output from "stream_out" and applies it to the specified
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
443 repository.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
444
30332
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29919
diff changeset
445 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29919
diff changeset
446 handled by this function.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
447 """
27859
f55a5ace8e69 with: use context manager in streamclone consumev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27850
diff changeset
448 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
449 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
450 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
451 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
452 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
453 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
454 _(b'clone'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
455 )
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
456 progress.update(0)
30975
22fbca1d11ed mercurial: switch to util.timer for all interval timings
Simon Farnsworth <simonfar@fb.com>
parents: 30332
diff changeset
457 start = util.timer()
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
458
29919
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
459 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
460 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
461 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
462 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
463 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
464 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
465 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
466 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
467 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
468 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
469 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
470 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
471 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
472
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
473 total_file_count = 0
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
474 with repo.transaction(b'clone'):
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
475 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
49284
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 49190
diff changeset
476 for i in range(filecount):
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
477 # XXX doesn't support '\n' or '\r' in filenames
50925
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50703
diff changeset
478 if hasattr(fp, 'readline'):
50663
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
479 l = fp.readline()
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
480 else:
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
481 # inline clonebundles use a chunkbuffer, so no readline
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
482 # --> this should be small anyway, the first line
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
483 # only contains the size of the bundle
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
484 l_buf = []
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
485 while not (l_buf and l_buf[-1] == b'\n'):
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
486 l_buf.append(fp.read(1))
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
487 l = b''.join(l_buf)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
488 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
489 name, size = l.split(b'\0', 1)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
490 size = int(size)
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
491 except (ValueError, TypeError):
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
492 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
493 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
494 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
495 if repo.ui.debugflag:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
496 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
497 b'adding %s (%s)\n' % (name, util.bytecount(size))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
498 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
499 # for backwards compat, name was partially encoded
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
500 path = store.decodedir(name)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
501 with repo.svfs(path, b'w', backgroundclose=True) as ofp:
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
502 total_file_count += 1
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
503 for chunk in util.filechunkiter(fp, limit=size):
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
504 progress.increment(step=len(chunk))
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
505 ofp.write(chunk)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
506
29919
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
507 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
508 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
509 repo.invalidate(clearfilecache=True)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
510
38373
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38360
diff changeset
511 progress.complete()
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
512 _report_transferred(repo, start, total_file_count, bytecount)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
513
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
514
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
515 def readbundle1header(fp) -> tuple[int, int, Set[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
516 compression = fp.read(2)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
517 if compression != b'UN':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
518 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
519 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
520 b'only uncompressed stream clone bundles are '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
521 b'supported; got %s'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
522 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
523 % compression
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
524 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
525
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
526 filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
527 requireslen = struct.unpack(b'>H', fp.read(2))[0]
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
528 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
529
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
530 if not requires.endswith(b'\0'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
531 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
532 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
533 b'malformed stream clone bundle: '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
534 b'requirements not properly encoded'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
535 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
536 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
537
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
538 requirements = set(requires.rstrip(b'\0').split(b','))
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
539
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
540 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
541
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
542
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
543 def applybundlev1(repo, fp) -> None:
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
544 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
545
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
546 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
547 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
548 """
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
549 if len(repo):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
550 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
551 _(b'cannot apply stream clone bundle on non-empty repo')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
552 )
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
553
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
554 filecount, bytecount, requirements = readbundle1header(fp)
48670
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
555 missingreqs = requirements - repo.supported
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
556 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
557 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
558 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
559 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
560 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
561
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
562 consumev1(repo, fp, filecount, bytecount)
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
563 nodemap.post_stream_cleanup(repo)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
564
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
565
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
566 class streamcloneapplier:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
567 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
568
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
569 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
570 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
571 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
572
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
573 def __init__(self, fh) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
574 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
575
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
576 def apply(self, repo) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
577 return applybundlev1(repo, self._fh)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
578
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
579
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
580 # type of file to stream
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
581 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
582 _filefull = 1 # full snapshot file
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
583
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
584 # Source of the file
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
585 _srcstore = b's' # store (svfs)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
586 _srccache = b'c' # cache (cache)
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
587
51699
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51696
diff changeset
588
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
589 # This is it's own function so extensions can override it.
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
590 def _walkstreamfullstorefiles(repo) -> list[bytes]:
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
591 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
592 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
593 if not repo.publishing():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
594 fnames.append(b'phaseroots')
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
595 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
596
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
597
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
598 def _filterfull(entry, copy, vfsmap):
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
599 """actually copy the snapshot files"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
600 src, name, ftype, data = entry
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
601 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
602 return entry
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
603 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
604
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
605
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51859
diff changeset
606 class VolatileManager:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
607 """Manage temporary backups of volatile files during stream clone.
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
608
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
609 This class will keep open file handles for the volatile files, writing the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
610 smaller ones on disk if the number of open file handles grow too much.
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
611
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
612 This should be used as a Python context, the file handles and copies will
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
613 be discarded when exiting the context.
49190
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48946
diff changeset
614
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
615 The preservation can be done by calling the object on the real path
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
616 (encoded full path).
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
617
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
618 Valid filehandles for any file should be retrieved by calling `open(path)`.
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
619 """
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
620
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
621 # arbitrarily picked as "it seemed fine" and much higher than the current
52184
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
622 # usage. The Windows value of 2 is actually 1 file open at a time, due to
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
623 # the `flush_count = self.MAX_OPEN // 2` and `self.MAX_OPEN - 1` threshold
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
624 # for flushing to disk in __call__().
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
625 MAX_OPEN = 2 if pycompat.iswindows else 100
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
626
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
627 def __init__(self) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
628 self._counter = 0
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
629 self._volatile_fps = None
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
630 self._copies = None
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
631 self._dst_dir = None
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
632
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
633 def __enter__(self):
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
634 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
635 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
636 self._volatile_fps = {}
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
637 self._counter += 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
638 return self
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
639
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
640 def __exit__(self, *args, **kwars):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
641 """discard all backups"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
642 self._counter -= 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
643 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
644 for _size, fp in self._volatile_fps.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
645 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
646 self._volatile_fps = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
647 if self._copies is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
648 for tmp in self._copies.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
649 util.tryunlink(tmp)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
650 util.tryrmdir(self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
651 self._copies = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
652 self._dst_dir = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
653 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
654 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
655 assert self._dst_dir is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
656
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
657 def _init_tmp_copies(self) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
658 """prepare a temporary directory to save volatile files
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
659
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
660 This will be used as backup if we have too many files open"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
661 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
662 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
663 assert self._dst_dir is None
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
664 self._copies = {}
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
665 self._dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
666
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
667 def _flush_some_on_disk(self) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
668 """move some of the open files to tempory files on disk"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
669 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
670 self._init_tmp_copies()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
671 flush_count = self.MAX_OPEN // 2
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
672 for src, (size, fp) in sorted(self._volatile_fps.items())[:flush_count]:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
673 prefix = os.path.basename(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
674 fd, dst = pycompat.mkstemp(prefix=prefix, dir=self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
675 self._copies[src] = dst
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
676 os.close(fd)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
677 # we no longer hardlink, but on the other hand we rarely do this,
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
678 # and we do it for the smallest file only and not at all in the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
679 # common case.
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
680 with open(dst, 'wb') as bck:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
681 fp.seek(0)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
682 bck.write(fp.read())
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
683 del self._volatile_fps[src]
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
684 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
685
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
686 def _keep_one(self, src: bytes) -> int:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
687 """preserve an open file handle for a given path"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
688 # store the file quickly to ensure we close it if any error happens
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
689 _, fp = self._volatile_fps[src] = (None, open(src, 'rb'))
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
690 fp.seek(0, os.SEEK_END)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
691 size = fp.tell()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
692 self._volatile_fps[src] = (size, fp)
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
693 return size
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
694
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
695 def __call__(self, src: bytes) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
696 """preserve the volatile file at src"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
697 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
698 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
699 self._flush_some_on_disk()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
700 self._keep_one(src)
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
701
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
702 def try_keep(self, src: bytes) -> Optional[int]:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
703 """record a volatile file and returns it size
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
704
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
705 return None if the file does not exists.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
706
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
707 Used for cache file that are not lock protected.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
708 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
709 assert 0 < self._counter
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
710 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
711 self._flush_some_on_disk()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
712 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
713 return self._keep_one(src)
52640
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52359
diff changeset
714 except OSError as err:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
715 if err.errno not in (errno.ENOENT, errno.EPERM):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
716 raise
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
717 return None
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
718
52013
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
719 @contextlib.contextmanager
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
720 def open(self, src):
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
721 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
722 entry = self._volatile_fps.get(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
723 if entry is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
724 _size, fp = entry
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
725 fp.seek(0)
52013
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
726 yield fp
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
727 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
728 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
729 actual_path = src
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
730 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
731 actual_path = self._copies.get(src, src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
732 with open(actual_path, 'rb') as fp:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
733 yield fp
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
734
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
735
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
736 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
737 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
738 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
739 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
740 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
741 }
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
742 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
743 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
744 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
745
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
746 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
747
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
748
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
749 def _emit2(repo, entries):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
750 """actually emit the stream bundle"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
751 vfsmap = _makemap(repo)
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
752 # we keep repo.vfs out of the on purpose, ther are too many danger there
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
753 # (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
754 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
755 # this assert is duplicated (from _makemap) as author might think this is
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
756 # fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
757 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
758 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
759 'repo.vfs must not be added to vfsmap for security reasons'
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
760 )
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
761
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
762 # translate the vfs one
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
763 entries = [(vfs_key, vfsmap[vfs_key], e) for (vfs_key, e) in entries]
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
764 _test_sync_point_walk_1_2(repo)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
765
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
766 max_linkrev = len(repo)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
767 file_count = totalfilesize = 0
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
768 with VolatileManager() as volatiles:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
769 # make sure we preserve volatile files
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
770 with util.nogc():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
771 # record the expected size of every file
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
772 for k, vfs, e in entries:
52357
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52356
diff changeset
773 e.preserve_volatiles(vfs, volatiles)
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
774 for f in e.files():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
775 file_count += 1
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
776 totalfilesize += f.file_size(vfs)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
777
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
778 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
779 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
780 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
781 progress.update(0)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
782 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
783 # the first yield release the lock on the repository
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
784 yield file_count, totalfilesize
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
785 totalbytecount = 0
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
786
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
787 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
788 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
789 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
790 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
791 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
792 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
793 preserve_file_count=True,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
794 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
795 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
796 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
797 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
798 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
799 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
800 bytecount = 0
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
801 for chunk in stream:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
802 bytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
803 totalbytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
804 progress.update(totalbytecount)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
805 yield chunk
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
806 if bytecount != size:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
807 # Would most likely be caused by a race due to `hg
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
808 # strip` or a revlog split
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
809 msg = _(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
810 b'clone could only read %d bytes from %s, but '
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
811 b'expected %d bytes'
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
812 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
813 raise error.Abort(msg % (bytecount, name, size))
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
814
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
815
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
816 def _emit3(repo, entries) -> Iterator[bytes | None]:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
817 """actually emit the stream bundle (v3)"""
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
818 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
819 # we keep repo.vfs out of the map on purpose, ther are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
820 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
821 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
822 # this assert is duplicated (from _makemap) as authors might think this is
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
823 # fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
824 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
825 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
826 'repo.vfs must not be added to vfsmap for security reasons'
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
827 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
828
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
829 # translate the vfs once
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
830 # we only turn this into a list for the `_test_sync`, this is not ideal
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
831 base_entries = list(entries)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
832 _test_sync_point_walk_1_2(repo)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
833 entries = []
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
834 with VolatileManager() as volatiles:
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51859
diff changeset
835 # make sure we preserve volatile files
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
836 for vfs_key, e in base_entries:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
837 vfs = vfsmap[vfs_key]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
838 any_files = True
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
839 if e.maybe_volatile:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
840 any_files = False
52357
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52356
diff changeset
841 e.preserve_volatiles(vfs, volatiles)
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
842 for f in e.files():
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
843 if f.is_volatile:
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
844 # record the expected size under lock
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
845 f.file_size(vfs)
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
846 any_files = True
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
847 if any_files:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
848 entries.append((vfs_key, vfsmap[vfs_key], e))
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
849
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
850 total_entry_count = len(entries)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
851
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
852 max_linkrev = len(repo)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
853 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
854 _(b'bundle'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
855 total=total_entry_count,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
856 unit=_(b'entry'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
857 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
858 progress.update(0)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
859 # the first yield release the lock on the repository
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
860 yield None
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
861 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
862 yield util.uvarintencode(total_entry_count)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
863
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
864 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
865 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
866 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
867 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
868 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
869 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
870 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
871 yield util.uvarintencode(len(entry_streams))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
872 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
873 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
874 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
875 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
876 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
877 yield from stream
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
878 progress.increment()
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
879
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
880
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
881 def _test_sync_point_walk_1_2(repo):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
882 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
883
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
884 Triggered after gather entry, but before starting to process/preserve them
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
885 under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
886
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
887 (on v1 is triggered before the actual walk start)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
888 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
889
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
890
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
891 def _test_sync_point_walk_3(repo):
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
892 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
893
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
894 Triggered right before releasing the lock, but after computing what need
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
895 needed to compute under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
896 """
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
897
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
898
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
899 def _test_sync_point_walk_4(repo):
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
900 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
901
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
902 Triggered right after releasing the lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
903 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
904
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
905
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
906 # not really a StoreEntry, but close enough
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
907 class CacheEntry(store.SimpleStoreEntry):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
908 """Represent an entry for Cache files
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
909
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
910 It has special logic to preserve cache file early and accept optional
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
911 presence.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
912
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
913
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
914 (Yes... this is not really a StoreEntry, but close enough. We could have a
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
915 BaseEntry base class, bbut the store one would be identical)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
916 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
917
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
918 def __init__(self, entry_path) -> None:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
919 super().__init__(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
920 entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
921 # we will directly deal with that in `setup_cache_file`
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
922 is_volatile=True,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
923 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
924
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
925 def preserve_volatiles(self, vfs, volatiles) -> None:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
926 self._file_size = volatiles.try_keep(vfs.join(self._entry_path))
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
927 if self._file_size is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
928 self._files = []
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
929 else:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
930 assert self._is_volatile
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
931 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
932 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
933 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
934 file_size=self._file_size,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
935 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
936 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
937 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
938
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
939 def files(self) -> list[store.StoreFile]:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
940 if self._files is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
941 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
942 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
943 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
944 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
945 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
946 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
947 return super().files()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
948
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
949
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
950 class CacheFile(store.StoreFile):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
951 # inform the "copy/hardlink" version that this file might be missing
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
952 # without consequences.
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
953 optional: bool = True
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
954
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
955
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
956 def _entries_walk(repo, includes, excludes, includeobsmarkers: bool):
50512
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
957 """emit a seris of files information useful to clone a repo
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
958
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
959 return (vfs-key, entry) iterator
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
960
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
961 Where `entry` is StoreEntry. (used even for cache entries)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
962 """
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
963 assert repo._currentlock(repo._lockref) is not None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
964
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
965 matcher = None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
966 if includes or excludes:
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
967 matcher = narrowspec.match(repo.root, includes, excludes)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
968
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
969 phase = not repo.publishing()
51547
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
970 # Python is getting crazy at all the small container we creates, disabling
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
971 # the gc while we do so helps performance a lot.
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
972 with util.nogc():
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
973 entries = _walkstreamfiles(
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
974 repo,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
975 matcher,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
976 phase=phase,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
977 obsolescence=includeobsmarkers,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
978 )
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
979 for entry in entries:
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
980 yield (_srcstore, entry)
50512
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
981
51547
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
982 for name in cacheutil.cachetocopy(repo):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
983 if repo.cachevfs.exists(name):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
984 # not really a StoreEntry, but close enough
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
985 yield (_srccache, CacheEntry(entry_path=name))
50513
43ed1f12b00a stream-clone: yield cache entry in `_entries_walk` too
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50512
diff changeset
986
50512
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
987
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
988 def generatev2(repo, includes, excludes, includeobsmarkers: bool):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
989 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
990
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
991 the data stream consists the following entries:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
992 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
993 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
994 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
995 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
996 5) N bytes containing the file data
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
997
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
998 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
999 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1000
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1001 with repo.lock():
47444
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1002 repo.ui.debug(b'scanning\n')
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
1003
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1004 entries = _entries_walk(
47444
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1005 repo,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1006 includes=includes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1007 excludes=excludes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1008 includeobsmarkers=includeobsmarkers,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1009 )
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1010 chunks = _emit2(repo, entries)
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
1011 first = next(chunks)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1012 file_count, total_file_size = first
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1013 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1014 _test_sync_point_walk_4(repo)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1015
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1016 return file_count, total_file_size, chunks
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1017
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1018
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1019 def generatev3(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1020 repo, includes, excludes, includeobsmarkers: bool
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1021 ) -> Iterator[bytes | None]:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1022 """Emit content for version 3 of a streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1023
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1024 the data stream consists the following:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1025 1) A varint E containing the number of entries (can be 0), then E entries follow
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1026 2) For each entry:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1027 2.1) The number of files in this entry (can be 0, but typically 1 or 2)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1028 2.2) For each file:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1029 2.2.1) A char representing the file destination (eg: store or cache)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1030 2.2.2) A varint N containing the length of the filename
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1031 2.2.3) A varint M containing the length of file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1032 2.2.4) N bytes containing the filename (the internal, store-agnostic form)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1033 2.2.5) M bytes containing the file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1034
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1035 Returns the data iterator.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1036
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1037 XXX This format is experimental and subject to change. Here is a
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1038 XXX non-exhaustive list of things this format could do or change:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1039
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1040 - making it easier to write files in parallel
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1041 - holding the lock for a shorter time
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1042 - improving progress information
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1043 - ways to adjust the number of expected entries/files ?
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1044 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1045
51548
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1046 # Python is getting crazy at all the small container we creates while
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1047 # considering the files to preserve, disabling the gc while we do so helps
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1048 # performance a lot.
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1049 with repo.lock(), util.nogc():
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1050 repo.ui.debug(b'scanning\n')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1051
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1052 entries = _entries_walk(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1053 repo,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1054 includes=includes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1055 excludes=excludes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1056 includeobsmarkers=includeobsmarkers,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1057 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1058 chunks = _emit3(repo, list(entries))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1059 first = next(chunks)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1060 assert first is None
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1061 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1062 _test_sync_point_walk_4(repo)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1063
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1064 return chunks
50521
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50520
diff changeset
1065
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50520
diff changeset
1066
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1067 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1068 def nested(*ctxs):
39757
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1069 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1070 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1071 with this:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1072 if rest:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1073 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1074 yield
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1075 else:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1076 yield
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1077
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1078
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1079 class V2Report:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1080 """a small class to track the data we saw within the stream"""
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1081
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1082 def __init__(self):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1083 self.byte_count = 0
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1084
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1085
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1086 def consumev2(repo, fp, filecount: int, filesize: int) -> None:
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1087 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1088
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1089 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1090 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1091 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1092 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1093 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1094 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1095 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1096 )
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1097 progress = repo.ui.makeprogress(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1098 _(b'clone'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1099 total=filesize,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1100 unit=_(b'bytes'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1101 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1102 start = util.timer()
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1103 report = V2Report()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1104
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1105 vfsmap = _makemap(repo)
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1106 # we keep repo.vfs out of the on purpose, ther are too many danger
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1107 # there (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1108 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1109 # this assert is duplicated (from _makemap) as author might think this
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1110 # is fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1111 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1112 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
1113 'repo.vfs must not be added to vfsmap for security reasons'
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1114 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1115
52913
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
1116 cpu_profile = scmutil.get_resource_profile(repo.ui, b'cpu')
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1117 threaded = repo.ui.configbool(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1118 b"worker", b"parallel-stream-bundle-processing"
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1119 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1120 num_writer = repo.ui.configint(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1121 b"worker",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1122 b"parallel-stream-bundle-processing.num-writer",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1123 )
52913
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
1124 if num_writer <= 0:
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
1125 num_writer = DEFAULT_NUM_WRITER[cpu_profile]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1126 with repo.transaction(b'clone'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1127 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1128 with nested(*ctxs):
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1129 workers = []
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1130 info_queue = None
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1131 data_queue = None
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1132 try:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1133 if not threaded:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1134 fc = _FileChunker
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1135 raw_data = fp
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1136 else:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1137 fc = _ThreadSafeFileChunker
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1138 data_queue = _DataQueue()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1139 raw_data = util.chunkbuffer(data_queue)
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1140
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1141 # XXX we will drop this extra filechunkiter layer soon
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1142 part_content = util.filechunkiter(fp)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1143 w = threading.Thread(
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1144 target=data_queue.fill_from,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1145 args=(part_content,),
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1146 )
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1147 workers.append(w)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1148 w.start()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1149 files = _v2_parse_files(
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1150 repo,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1151 raw_data,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1152 filecount,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1153 progress,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1154 report,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1155 file_chunker=fc,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1156 )
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1157 if not threaded:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1158 _write_files(vfsmap, files)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1159 else:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1160 info_queue = _FileInfoQueue(files)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1161
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1162 for __ in range(num_writer):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1163 w = threading.Thread(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1164 target=_write_files,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1165 args=(vfsmap, info_queue),
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1166 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1167 workers.append(w)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1168 w.start()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1169 info_queue.fill()
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1170 except: # re-raises
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1171 if data_queue is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1172 data_queue.abort()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1173 raise
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1174 finally:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1175 # shut down all the workers
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1176 if info_queue is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1177 # this is strictly speaking one too many worker for
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1178 # this queu, but closing too many is not a problem.
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1179 info_queue.close(len(workers))
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1180 for w in workers:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1181 w.join()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1182
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1183 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1184 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1185 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1186
38373
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38360
diff changeset
1187 progress.complete()
52703
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1188 # acknowledge the end of the bundle2 part, this help aligning
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1189 # sequential and parallel behavior.
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1190 remains = fp.read(1)
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1191 assert not remains
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1192 _report_transferred(repo, start, filecount, report.byte_count)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1193
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1194
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1195 # iterator of chunk of bytes that constitute a file content.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1196 FileChunksT = Iterator[bytes]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1197 # Contains the information necessary to write stream file on disk
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1198 FileInfoT = Tuple[
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1199 bytes, # vfs key
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1200 bytes, # file name (non-vfs-encoded)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1201 FileChunksT, # content
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1202 ]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1203
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1204
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1205 class _DataQueue:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1206 """A queue passing data from the bundle stream to other thread"""
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1207
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1208 def __init__(self):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1209 self._q = queue.Queue()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1210 self._abort = False
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1211
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1212 def fill_from(self, data):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1213 """fill the data queue from a bundle2 part object
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1214
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1215 This is meant to be called by the data reading thread
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1216 """
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1217 q = self._q
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1218 try:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1219 for item in data:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1220 q.put(item)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1221 if self._abort:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1222 break
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1223 finally:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1224 q.put(None)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1225
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1226 def __iter__(self):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1227 """Iterate over the bundle chunkgs
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1228
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1229 This is meant to be called by the data parsing thread."""
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1230 q = self._q
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1231 while (i := q.get()) is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1232 yield i
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1233 if self._abort:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1234 break
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1235
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1236 def abort(self):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1237 """stop the data-reading thread and interrup the iteration of the consumer
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1238
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1239 This is meant to be called on error.
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1240 """
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1241 self._abort = True
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1242 self._q.put(None)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1243
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1244
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1245 class _FileInfoQueue:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1246 """A thread-safe queue to passer parsed file information to the writers"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1247
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1248 def __init__(self, info: Iterable[FileInfoT]):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1249 self._info = info
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1250 self._q = queue.Queue()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1251
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1252 def fill(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1253 """iterate over the parsed information to file the queue
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1254
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1255 This is meant to be call from the thread parsing the stream information.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1256 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1257 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1258 for i in self._info:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1259 q.put(i)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1260
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1261 def close(self, number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1262 """signal all the workers that we no longer have any file info coming
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1263
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1264 Called from the thread parsing the stream information (and/or the main
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1265 thread if different).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1266 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1267 for __ in range(number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1268 self._q.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1269
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1270 def __iter__(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1271 """iterate over the available file info
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1272
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1273 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1274 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1275 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1276 while (i := q.get()) is not None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1277 yield i
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1278
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1279
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1280 class _FileChunker:
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1281 """yield the chunk that constitute a file
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1282
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1283 This class exists as the counterpart of the threaded version and
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1284 would not be very useful on its own.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1285 """
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1286
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1287 def __init__(
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1288 self,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1289 fp: bundle2mod.unbundlepart,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1290 data_len: int,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1291 progress: scmutil.progress,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1292 report: V2Report,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1293 ):
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1294 self.report = report
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1295 self.progress = progress
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1296 self._chunks = util.filechunkiter(fp, limit=data_len)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1297
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1298 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1299 """Do nothing in non-threading context"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1300
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1301 def __iter__(self) -> FileChunksT:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1302 for chunk in self._chunks:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1303 self.report.byte_count += len(chunk)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1304 self.progress.increment(step=len(chunk))
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1305 yield chunk
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1306
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1307
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1308 class _ThreadSafeFileChunker(_FileChunker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1309 """yield the chunk that constitute a file
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1310
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1311 Make sure you call the "fill" function in the main thread to read the
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1312 right data at the right time.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1313 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1314
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1315 def __init__(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1316 self,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1317 fp: bundle2mod.unbundlepart,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1318 data_len: int,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1319 progress: scmutil.progress,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1320 report: V2Report,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1321 ):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1322 super().__init__(fp, data_len, progress, report)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1323 self._queue = queue.Queue()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1324
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1325 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1326 """fill the file chunker queue with data read from the stream
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1327
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1328 This is meant to be called from the thread parsing information (and
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1329 consuming the stream data).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1330 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1331 try:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1332 for chunk in super().__iter__():
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1333 self._queue.put(chunk)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1334 finally:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1335 self._queue.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1336
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1337 def __iter__(self) -> FileChunksT:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1338 """Iterate over all the file chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1339
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1340 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1341 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1342 while (chunk := self._queue.get()) is not None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1343 yield chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1344
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1345
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1346 def _v2_parse_files(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1347 repo,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1348 fp: bundle2mod.unbundlepart,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1349 file_count: int,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1350 progress: scmutil.progress,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1351 report: V2Report,
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1352 file_chunker: Type[_FileChunker] = _FileChunker,
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1353 ) -> Iterator[FileInfoT]:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1354 """do the "stream-parsing" part of stream v2
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1355
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1356 The parsed information are yield result for consumption by the "writer"
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1357 """
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1358 progress.update(0)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1359 for i in range(file_count):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1360 src = util.readexactly(fp, 1)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1361 namelen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1362 datalen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1363
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1364 name = util.readexactly(fp, namelen)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1365
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1366 if repo.ui.debugflag:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1367 repo.ui.debug(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1368 b'adding [%s] %s (%s)\n' % (src, name, util.bytecount(datalen))
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1369 )
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1370 chunks = file_chunker(fp, datalen, progress, report)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1371 yield (src, name, iter(chunks))
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1372 # make sure we read all the chunk before moving to the next file
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1373 chunks.fill()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1374
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1375
52910
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1376 def _write_files(vfsmap, info: Iterable[FileInfoT]):
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1377 """write files from parsed data"""
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1378 for src, name, data in info:
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1379 vfs = vfsmap[src]
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1380 with vfs(name, b'w') as ofp:
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1381 for chunk in data:
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1382 ofp.write(chunk)
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1383
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1384
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1385 def consumev3(repo, fp) -> None:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1386 """Apply the contents from a version 3 streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1387
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1388 Data is read from an object that only needs to provide a ``read(size)``
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1389 method.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1390 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1391 with repo.lock():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1392 start = util.timer()
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1393
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1394 entrycount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1395 repo.ui.status(_(b'%d entries to transfer\n') % (entrycount))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1396
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1397 progress = repo.ui.makeprogress(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1398 _(b'clone'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1399 total=entrycount,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1400 unit=_(b'entries'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1401 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1402 progress.update(0)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1403 bytes_transferred = 0
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1404
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1405 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1406 # we keep repo.vfs out of the on purpose, there are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1407 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1408 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1409 # this assert is duplicated (from _makemap) as authors might think this
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1410 # is fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1411 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1412 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
1413 'repo.vfs must not be added to vfsmap for security reasons'
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1414 )
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
1415 total_file_count = 0
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1416 with repo.transaction(b'clone'):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1417 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1418 with nested(*ctxs):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1419 for i in range(entrycount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1420 filecount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1421 if filecount == 0:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1422 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1423 repo.ui.debug(b'entry with no files [%d]\n' % (i))
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
1424 total_file_count += filecount
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1425 for i in range(filecount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1426 src = util.readexactly(fp, 1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1427 vfs = vfsmap[src]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1428 namelen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1429 datalen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1430
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1431 name = util.readexactly(fp, namelen)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1432
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1433 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1434 msg = b'adding [%s] %s (%s)\n'
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1435 msg %= (src, name, util.bytecount(datalen))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1436 repo.ui.debug(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1437 bytes_transferred += datalen
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1438
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1439 with vfs(name, b'w') as ofp:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1440 for chunk in util.filechunkiter(fp, limit=datalen):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1441 ofp.write(chunk)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1442 progress.increment(step=1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1443
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1444 # force @filecache properties to be reloaded from
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1445 # streamclone-ed file at next access
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1446 repo.invalidate(clearfilecache=True)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1447
52701
6feb3b3029b5 stream: consistently close progress before reporting time
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52700
diff changeset
1448 progress.complete()
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
1449 _report_transferred(repo, start, total_file_count, bytes_transferred)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1450
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1451
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1452 def applybundlev2(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1453 repo, fp, filecount: int, filesize: int, requirements: Iterable[bytes]
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1454 ) -> None:
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
1455 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
1456
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1457 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1458 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1459 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
1460 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1461 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1462 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1463
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1464 consumev2(repo, fp, filecount, filesize)
35804
2d3e486d09d0 streamclone: move requirement update into consumev2
Boris Feld <boris.feld@octobus.net>
parents: 35803
diff changeset
1465
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
1466 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
1467 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
1468 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1469 )
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
1470 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1471 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1472 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
1473 scmutil.writereporequirements(repo)
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
1474 nodemap.post_stream_cleanup(repo)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1475
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1476
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1477 def applybundlev3(repo, fp, requirements: Iterable[bytes]) -> None:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1478 from . import localrepo
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1479
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1480 missingreqs = [r for r in requirements if r not in repo.supported]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1481 if missingreqs:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1482 msg = _(b'unable to apply stream clone: unsupported format: %s')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1483 msg %= b', '.join(sorted(missingreqs))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1484 raise error.Abort(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1485
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1486 consumev3(repo, fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1487
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1488 repo.requirements = new_stream_clone_requirements(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1489 repo.requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1490 requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1491 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1492 repo.svfs.options = localrepo.resolvestorevfsoptions(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1493 repo.ui, repo.requirements, repo.features
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1494 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1495 scmutil.writereporequirements(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1496 nodemap.post_stream_cleanup(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1497
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1498
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1499 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress) -> bool:
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1500 hardlink = [True]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1501
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1502 def copy_used():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1503 hardlink[0] = False
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1504 progress.topic = _(b'copying')
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1505
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1506 for k, path, optional in entries:
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1507 src_vfs = src_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1508 dst_vfs = dst_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1509 src_path = src_vfs.join(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1510 dst_path = dst_vfs.join(path)
47871
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1511 # We cannot use dirname and makedirs of dst_vfs here because the store
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1512 # encoding confuses them. See issue 6581 for details.
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1513 dirname = os.path.dirname(dst_path)
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1514 if not os.path.exists(dirname):
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1515 util.makedirs(dirname)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1516 dst_vfs.register_file(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1517 # XXX we could use the #nb_bytes argument.
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1518 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1519 util.copyfile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1520 src_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1521 dst_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1522 hardlink=hardlink[0],
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1523 no_hardlink_cb=copy_used,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1524 check_fs_hardlink=False,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1525 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1526 except FileNotFoundError:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1527 if not optional:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1528 raise
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1529 progress.increment()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1530 return hardlink[0]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1531
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1532
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1533 def local_copy(src_repo, dest_repo) -> None:
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1534 """copy all content from one local repository to another
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1535
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1536 This is useful for local clone"""
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1537 src_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1538 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1539 for r in src_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1540 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1541 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1542 dest_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1543 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1544 for r in dest_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1545 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1546 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1547 assert src_store_requirements == dest_store_requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1548
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1549 with dest_repo.lock():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1550 with src_repo.lock():
47448
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1551 # bookmark is not integrated to the streaming as it might use the
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1552 # `repo.vfs` and they are too many sentitive data accessible
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1553 # through `repo.vfs` to expose it to streaming clone.
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1554 src_book_vfs = bookmarks.bookmarksvfs(src_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1555 srcbookmarks = src_book_vfs.join(b'bookmarks')
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1556 bm_count = 0
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1557 if os.path.exists(srcbookmarks):
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1558 bm_count = 1
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1559
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1560 entries = _entries_walk(
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1561 src_repo,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1562 includes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1563 excludes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1564 includeobsmarkers=True,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1565 )
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1566 entries = list(entries)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1567 src_vfs_map = _makemap(src_repo)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1568 dest_vfs_map = _makemap(dest_repo)
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1569 total_files = sum(len(e[1].files()) for e in entries) + bm_count
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1570 progress = src_repo.ui.makeprogress(
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1571 topic=_(b'linking'),
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1572 total=total_files,
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1573 unit=_(b'files'),
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1574 )
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1575 # copy files
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1576 #
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1577 # We could copy the full file while the source repository is locked
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1578 # and the other one without the lock. However, in the linking case,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1579 # this would also requires checks that nobody is appending any data
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1580 # to the files while we do the clone, so this is not done yet. We
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1581 # could do this blindly when copying files.
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1582 files = [
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1583 (vfs_key, f.unencoded_path, f.optional)
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1584 for vfs_key, e in entries
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1585 for f in e.files()
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1586 ]
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1587 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1588
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1589 # copy bookmarks over
47448
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1590 if bm_count:
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1591 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1592 dstbookmarks = dst_book_vfs.join(b'bookmarks')
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1593 util.copyfile(srcbookmarks, dstbookmarks)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1594 progress.complete()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1595 if hardlink:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1596 msg = b'linked %d files\n'
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1597 else:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1598 msg = b'copied %d files\n'
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1599 src_repo.ui.debug(msg % total_files)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1600
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1601 with dest_repo.transaction(b"localclone") as tr:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1602 dest_repo.store.write(tr)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1603
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1604 # clean up transaction file as they do not make sense
50288
d89eecf9605e undo-files: no longer pass the `repo` to `cleanup_undo_files`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50286
diff changeset
1605 transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)