Mercurial > public > mercurial-scm > hg

annotate mercurial/streamclone.py @ 52913:58baa86c7a02

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
stream-clone-v2: make the number of writer dependent of usage config The current value are a bit arbitrary. I don't believe they will be the final value, but this changesets make sure some value exists, and we can adjust the actual value before graduating the feature from experimental.
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Mon, 27 Jan 2025 19:15:39 +0100
parents 7f848cfc4286
children d5ae681834e8
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
51859
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
8 from __future__ import annotations
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
10 import contextlib
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
11 import errno
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
12 import os
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
13 import queue
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
14 import struct
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
15 import threading
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
16
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
17 from typing import (
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
18 Iterable,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
19 Iterator,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
20 Optional,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
21 Set,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
22 Tuple,
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
23 Type,
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
24 )
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
25
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
26 from .i18n import _
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
27 from .interfaces import repository
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
28 from . import (
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
29 bookmarks,
50520
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
30 bundle2 as bundle2mod,
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
31 cacheutil,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
32 error,
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
33 narrowspec,
32744
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
34 phases,
38164
aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path
Yuya Nishihara <yuya@tcha.org>
parents: 37638
diff changeset
35 pycompat,
46626
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
36 requirements as requirementsmod,
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
37 scmutil,
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
38 store,
50286
3d0b5760851c undo-files: move the undo cleanup code in the transaction module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50279
diff changeset
39 transaction,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
40 util,
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
41 )
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
42 from .revlogutils import (
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
43 nodemap,
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
44 )
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
45
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
46
52913
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
47 # Number arbitrarily picked, feel free to change them (but the LOW one)
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
48 #
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
49 # update the configuration documentation if you touch this.
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
50 DEFAULT_NUM_WRITER = {
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
51 scmutil.RESOURCE_LOW: 1,
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
52 scmutil.RESOURCE_MEDIUM: 2,
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
53 scmutil.RESOURCE_HIGH: 4,
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
54 }
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
55
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
56
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
57 def new_stream_clone_requirements(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
58 default_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
59 streamed_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
60 ) -> Set[bytes]:
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
61 """determine the final set of requirement for a new stream clone
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
62
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
63 this method combine the "default" requirements that a new repository would
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
64 use with the constaint we get from the stream clone content. We keep local
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
65 configuration choice when possible.
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
66 """
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
67 requirements = set(default_requirements)
48601
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
68 requirements -= requirementsmod.STREAM_FIXED_REQUIREMENTS
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
69 requirements.update(streamed_requirements)
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
70 return requirements
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
71
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
72
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
73 def streamed_requirements(repo) -> Set[bytes]:
48597
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
74 """the set of requirement the new clone will have to support
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
75
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
76 This is used for advertising the stream options and to generate the actual
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
77 stream content."""
48601
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
78 requiredformats = (
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
79 repo.requirements & requirementsmod.STREAM_FIXED_REQUIREMENTS
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
80 )
48597
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
81 return requiredformats
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
82
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
83
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
84 def canperformstreamclone(pullop, bundle2: bool = False):
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
85 """Whether it is possible to perform a streaming clone as part of pull.
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
86
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
87 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
88 bundle2 and only through bundle2.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
89
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
90 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
91 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
92 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
93 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
94 """
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
95 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
96 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
97
50519
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
98 # should we consider streaming clone at all ?
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
99 streamrequested = pullop.streamclonerequested
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
100 # If we don't have a preference, let the server decide for us. This
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
101 # likely only comes into play in LANs.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
102 if streamrequested is None:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
103 # The server can advertise whether to prefer streaming clone.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
104 streamrequested = remote.capable(b'stream-preferred')
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
105 if not streamrequested:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
106 return False, None
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
107
50517
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
108 # Streaming clone only works on an empty destination repository
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
109 if len(repo):
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
110 return False, None
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
111
50518
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
112 # Streaming clone only works if all data is being requested.
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
113 if pullop.heads:
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
114 return False, None
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
115
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
116 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
117 if pullop.canusebundle2:
50520
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
118 local_caps = bundle2mod.getrepocaps(repo, role=b'client')
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
119 local_supported = set(local_caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
120 remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
121 bundle2supported = bool(local_supported & remote_supported)
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
122 # else
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
123 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
124 # the versions we support. Fall back and possibly allow legacy.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
125
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
126 # Ensures legacy code path uses available bundle2.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
127 if bundle2supported and not bundle2:
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
128 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
129 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
130 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
131 return False, None
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
132
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
133 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
134 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
135 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
136 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
137 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
138 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
139 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
140 requirements = set()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
141 if remote.capable(b'stream'):
46626
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
142 requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
143 else:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
144 streamreqs = remote.capable(b'streamreqs')
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
145 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
146 if not streamreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
147 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
148 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
149 b'warning: stream clone requested but server has them '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
150 b'disabled\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
151 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
152 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
153 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
154
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
155 streamreqs = set(streamreqs.split(b','))
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
156 # Server requires something we don't support. Bail.
48670
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
157 missingreqs = streamreqs - repo.supported
32259
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30975
diff changeset
158 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30975
diff changeset
159 pullop.repo.ui.warn(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
160 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
161 b'warning: stream clone requested but client is missing '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
162 b'requirements: %s\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
163 )
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
164 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
165 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
166 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
167 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
168 b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
169 b'for more information)\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
170 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
171 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
172 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
173 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
174
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
175 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
176
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
177
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
178 def maybeperformlegacystreamclone(pullop) -> None:
26462
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
179 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
180
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
181 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
182 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
183
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
184 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
185 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
186 """
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
187 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
188
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
189 supported, requirements = canperformstreamclone(pullop)
26458
362793295640 streamclone: refactor maybeperformstreamclone to take a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26447
diff changeset
190
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
191 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
192 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
193
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
194 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
195 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
196
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
197 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
198 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
199 rbranchmap = None
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
200 if remote.capable(b'branchmap'):
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
201 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
202 rbranchmap = e.callcommand(b'branchmap', {}).result()
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
203
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
204 repo.ui.status(_(b'streaming all changes\n'))
26470
4b5647d9ee13 streamclone: move "streaming all changes" message location
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26469
diff changeset
205
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
206 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
207 fp = e.callcommand(b'stream_out', {}).result()
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
208
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
209 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
210 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
211 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
212 # doesn't matter.
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
213 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
214 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
215 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
216 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
217 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
218 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
219 )
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
220 if resp == 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
221 raise error.Abort(_(b'operation forbidden by server'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
222 elif resp == 2:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
223 raise error.Abort(_(b'locking the remote repository failed'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
224 elif resp != 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
225 raise error.Abort(_(b'the server sent an unknown error code'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
226
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
227 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
228 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
229 filecount, bytecount = map(int, l.split(b' ', 1))
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
230 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
231 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
232 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
233 )
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
234
27850
49cfddbf54ba with: use context manager in maybeperformlegacystreamclone
Bryan O'Sullivan <bryano@fb.com>
parents: 27845
diff changeset
235 with repo.lock():
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
236 consumev1(repo, fp, filecount, bytecount)
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
237 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
238 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
239 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
240 )
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
241 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
242 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
243 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
244 scmutil.writereporequirements(repo)
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
245 nodemap.post_stream_cleanup(repo)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
246
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
247 if rbranchmap:
41615
328ca3b9e545 branchmap: encapsulate cache updating in the map itself
Martijn Pieters <mj@octobus.net>
parents: 40494
diff changeset
248 repo._branchcaches.replace(repo, rbranchmap)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
249
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
250 repo.invalidate()
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
251
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
252
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
253 def allowservergeneration(repo) -> bool:
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
254 """Whether streaming clones are allowed from the server."""
40028
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
255 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
256 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
257
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
258 if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
32744
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
259 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
260
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
261 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
262 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
263 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
264 if secret:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
265 return repo.ui.configbool(b'server', b'uncompressedallowsecret')
32744
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
266
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
267 return True
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
268
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
269
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
270 # This is it's own function so extensions can override it.
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
271 def _walkstreamfiles(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
272 repo, matcher=None, phase: bool = False, obsolescence: bool = False
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
273 ):
50511
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50509
diff changeset
274 return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
275
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
276
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
277 def _report_transferred(
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
278 repo, start_time: float, file_count: int, byte_count: int
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
279 ):
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
280 """common utility to report time it took to apply the stream bundle"""
52700
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
281 elapsed = util.timer() - start_time
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
282 if elapsed <= 0:
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
283 elapsed = 0.001
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
284 m = _(b'stream-cloned %d files / %s in %.1f seconds (%s/sec)\n')
52700
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
285 m %= (
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
286 file_count,
52700
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
287 util.bytecount(byte_count),
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
288 elapsed,
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
289 util.bytecount(byte_count / elapsed),
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
290 )
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
291 repo.ui.status(m)
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
292
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
293
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
294 def generatev1(repo) -> tuple[int, int, Iterator[bytes]]:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
295 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
296
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
297 This returns a 3-tuple of (file count, byte size, data iterator).
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
298
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
299 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
300 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
301 delimited by a null byte.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
302
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
303 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
304 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
305
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
306 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
307 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
308 for adding it.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
309
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
310 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
311 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
312 """
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
313 entries = []
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
314 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
315 # Get consistent snapshot of repo, lock during scan.
27845
7417e1c10253 with: use context manager in streamclone generatev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27794
diff changeset
316 with repo.lock():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
317 repo.ui.debug(b'scanning\n')
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
318 _test_sync_point_walk_1_2(repo)
50468
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50288
diff changeset
319 for entry in _walkstreamfiles(repo):
50469
9fdc28e21b68 store: introduce a EntryFile object to actually access file info
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50468
diff changeset
320 for f in entry.files():
50474
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
321 file_size = f.file_size(repo.store.vfs)
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
322 if file_size:
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
323 entries.append((f.unencoded_path, file_size))
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
324 total_bytes += file_size
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
325 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
326 _test_sync_point_walk_4(repo)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
327
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
328 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
329 b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
330 )
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
331
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
332 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
333 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
334
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
335 def emitrevlogdata() -> Iterator[bytes]:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
336 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
337 if debugflag:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
338 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
339 # partially encode name over the wire for backwards compat
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
340 yield b'%s\0%d\n' % (store.encodedir(name), size)
33411
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
341 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
342 # trusted by the local repo) and expensive
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
343 with svfs(name, b'rb', auditpath=False) as fp:
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
344 if size <= 65536:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
345 yield fp.read(size)
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
346 else:
52644
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52640
diff changeset
347 yield from util.filechunkiter(fp, limit=size)
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
348
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
349 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
350
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
351
52691
279e217d6041 typing: lock in the new type annotations detected with the pyupgrade changes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52644
diff changeset
352 def generatev1wireproto(repo) -> Iterator[bytes]:
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
353 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
354
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
355 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
356 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
357 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
358
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
359 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
360 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
361 a permissions error for the server process).
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
362 """
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
363 if not allowservergeneration(repo):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
364 yield b'1\n'
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
365 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
366
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
367 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
368 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
369 except error.LockError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
370 yield b'2\n'
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
371 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
372
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
373 # Indicates successful response.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
374 yield b'0\n'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
375 yield b'%d %d\n' % (filecount, bytecount)
52644
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52640
diff changeset
376 yield from it
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
377
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
378
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
379 def generatebundlev1(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
380 repo, compression: bytes = b'UN'
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
381 ) -> tuple[Set[bytes], Iterator[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
382 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
383
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
384 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
385 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
386
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
387 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
388 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
389
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
390 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
391 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
392
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
393 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
394 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
395 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
396 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
397
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
398 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
399 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
400
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
401 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
402 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
403 if compression != b'UN':
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
404 raise ValueError('we do not support the compression argument yet')
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
405
48597
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
406 requirements = streamed_requirements(repo)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
407 requires = b','.join(sorted(requirements))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
408
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
409 def gen() -> Iterator[bytes]:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
410 yield b'HGS1'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
411 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
412
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
413 filecount, bytecount, it = generatev1(repo)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
414 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
415 _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
416 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
417
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
418 yield struct.pack(b'>QQ', filecount, bytecount)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
419 yield struct.pack(b'>H', len(requires) + 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
420 yield requires + b'\0'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
421
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
422 # This is where we'll add compression in the future.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
423 assert compression == b'UN'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
424
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
425 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
426 _(b'bundle'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
427 )
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
428 progress.update(0)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
429
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
430 for chunk in it:
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
431 progress.increment(step=len(chunk))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
432 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
433
38373
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38360
diff changeset
434 progress.complete()
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
435
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
436 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
437
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
438
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
439 def consumev1(repo, fp, filecount: int, bytecount: int) -> None:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
440 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
441
30332
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29919
diff changeset
442 This takes the output from "stream_out" and applies it to the specified
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
443 repository.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
444
30332
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29919
diff changeset
445 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29919
diff changeset
446 handled by this function.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
447 """
27859
f55a5ace8e69 with: use context manager in streamclone consumev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27850
diff changeset
448 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
449 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
450 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
451 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
452 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
453 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
454 _(b'clone'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
455 )
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
456 progress.update(0)
30975
22fbca1d11ed mercurial: switch to util.timer for all interval timings
Simon Farnsworth <simonfar@fb.com>
parents: 30332
diff changeset
457 start = util.timer()
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
458
29919
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
459 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
460 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
461 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
462 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
463 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
464 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
465 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
466 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
467 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
468 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
469 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
470 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
471 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
472
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
473 total_file_count = 0
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
474 with repo.transaction(b'clone'):
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
475 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
49284
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 49190
diff changeset
476 for i in range(filecount):
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
477 # XXX doesn't support '\n' or '\r' in filenames
50925
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50703
diff changeset
478 if hasattr(fp, 'readline'):
50663
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
479 l = fp.readline()
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
480 else:
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
481 # inline clonebundles use a chunkbuffer, so no readline
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
482 # --> this should be small anyway, the first line
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
483 # only contains the size of the bundle
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
484 l_buf = []
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
485 while not (l_buf and l_buf[-1] == b'\n'):
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
486 l_buf.append(fp.read(1))
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
487 l = b''.join(l_buf)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
488 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
489 name, size = l.split(b'\0', 1)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
490 size = int(size)
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
491 except (ValueError, TypeError):
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
492 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
493 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
494 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
495 if repo.ui.debugflag:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
496 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
497 b'adding %s (%s)\n' % (name, util.bytecount(size))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
498 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
499 # for backwards compat, name was partially encoded
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
500 path = store.decodedir(name)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
501 with repo.svfs(path, b'w', backgroundclose=True) as ofp:
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
502 total_file_count += 1
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
503 for chunk in util.filechunkiter(fp, limit=size):
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
504 progress.increment(step=len(chunk))
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
505 ofp.write(chunk)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
506
29919
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
507 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
508 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
509 repo.invalidate(clearfilecache=True)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
510
38373
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38360
diff changeset
511 progress.complete()
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
512 _report_transferred(repo, start, total_file_count, bytecount)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
513
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
514
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
515 def readbundle1header(fp) -> tuple[int, int, Set[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
516 compression = fp.read(2)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
517 if compression != b'UN':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
518 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
519 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
520 b'only uncompressed stream clone bundles are '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
521 b'supported; got %s'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
522 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
523 % compression
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
524 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
525
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
526 filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
527 requireslen = struct.unpack(b'>H', fp.read(2))[0]
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
528 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
529
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
530 if not requires.endswith(b'\0'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
531 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
532 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
533 b'malformed stream clone bundle: '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
534 b'requirements not properly encoded'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
535 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
536 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
537
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
538 requirements = set(requires.rstrip(b'\0').split(b','))
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
539
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
540 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
541
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
542
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
543 def applybundlev1(repo, fp) -> None:
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
544 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
545
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
546 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
547 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
548 """
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
549 if len(repo):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
550 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
551 _(b'cannot apply stream clone bundle on non-empty repo')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
552 )
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
553
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
554 filecount, bytecount, requirements = readbundle1header(fp)
48670
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
555 missingreqs = requirements - repo.supported
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
556 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
557 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
558 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
559 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
560 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
561
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
562 consumev1(repo, fp, filecount, bytecount)
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
563 nodemap.post_stream_cleanup(repo)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
564
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
565
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
566 class streamcloneapplier:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
567 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
568
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
569 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
570 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
571 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
572
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
573 def __init__(self, fh) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
574 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
575
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
576 def apply(self, repo) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
577 return applybundlev1(repo, self._fh)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
578
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
579
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
580 # type of file to stream
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
581 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
582 _filefull = 1 # full snapshot file
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
583
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
584 # Source of the file
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
585 _srcstore = b's' # store (svfs)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
586 _srccache = b'c' # cache (cache)
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
587
51699
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51696
diff changeset
588
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
589 # This is it's own function so extensions can override it.
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
590 def _walkstreamfullstorefiles(repo) -> list[bytes]:
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
591 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
592 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
593 if not repo.publishing():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
594 fnames.append(b'phaseroots')
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
595 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
596
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
597
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
598 def _filterfull(entry, copy, vfsmap):
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
599 """actually copy the snapshot files"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
600 src, name, ftype, data = entry
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
601 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
602 return entry
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
603 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
604
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
605
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51859
diff changeset
606 class VolatileManager:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
607 """Manage temporary backups of volatile files during stream clone.
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
608
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
609 This class will keep open file handles for the volatile files, writing the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
610 smaller ones on disk if the number of open file handles grow too much.
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
611
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
612 This should be used as a Python context, the file handles and copies will
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
613 be discarded when exiting the context.
49190
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48946
diff changeset
614
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
615 The preservation can be done by calling the object on the real path
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
616 (encoded full path).
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
617
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
618 Valid filehandles for any file should be retrieved by calling `open(path)`.
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
619 """
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
620
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
621 # arbitrarily picked as "it seemed fine" and much higher than the current
52184
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
622 # usage. The Windows value of 2 is actually 1 file open at a time, due to
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
623 # the `flush_count = self.MAX_OPEN // 2` and `self.MAX_OPEN - 1` threshold
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
624 # for flushing to disk in __call__().
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
625 MAX_OPEN = 2 if pycompat.iswindows else 100
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
626
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
627 def __init__(self) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
628 self._counter = 0
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
629 self._volatile_fps = None
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
630 self._copies = None
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
631 self._dst_dir = None
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
632
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
633 def __enter__(self):
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
634 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
635 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
636 self._volatile_fps = {}
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
637 self._counter += 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
638 return self
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
639
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
640 def __exit__(self, *args, **kwars):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
641 """discard all backups"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
642 self._counter -= 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
643 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
644 for _size, fp in self._volatile_fps.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
645 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
646 self._volatile_fps = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
647 if self._copies is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
648 for tmp in self._copies.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
649 util.tryunlink(tmp)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
650 util.tryrmdir(self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
651 self._copies = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
652 self._dst_dir = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
653 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
654 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
655 assert self._dst_dir is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
656
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
657 def _init_tmp_copies(self) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
658 """prepare a temporary directory to save volatile files
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
659
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
660 This will be used as backup if we have too many files open"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
661 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
662 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
663 assert self._dst_dir is None
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
664 self._copies = {}
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
665 self._dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
666
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
667 def _flush_some_on_disk(self) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
668 """move some of the open files to tempory files on disk"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
669 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
670 self._init_tmp_copies()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
671 flush_count = self.MAX_OPEN // 2
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
672 for src, (size, fp) in sorted(self._volatile_fps.items())[:flush_count]:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
673 prefix = os.path.basename(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
674 fd, dst = pycompat.mkstemp(prefix=prefix, dir=self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
675 self._copies[src] = dst
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
676 os.close(fd)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
677 # we no longer hardlink, but on the other hand we rarely do this,
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
678 # and we do it for the smallest file only and not at all in the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
679 # common case.
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
680 with open(dst, 'wb') as bck:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
681 fp.seek(0)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
682 bck.write(fp.read())
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
683 del self._volatile_fps[src]
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
684 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
685
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
686 def _keep_one(self, src: bytes) -> int:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
687 """preserve an open file handle for a given path"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
688 # store the file quickly to ensure we close it if any error happens
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
689 _, fp = self._volatile_fps[src] = (None, open(src, 'rb'))
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
690 fp.seek(0, os.SEEK_END)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
691 size = fp.tell()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
692 self._volatile_fps[src] = (size, fp)
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
693 return size
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
694
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
695 def __call__(self, src: bytes) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
696 """preserve the volatile file at src"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
697 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
698 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
699 self._flush_some_on_disk()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
700 self._keep_one(src)
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
701
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
702 def try_keep(self, src: bytes) -> Optional[int]:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
703 """record a volatile file and returns it size
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
704
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
705 return None if the file does not exists.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
706
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
707 Used for cache file that are not lock protected.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
708 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
709 assert 0 < self._counter
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
710 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
711 self._flush_some_on_disk()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
712 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
713 return self._keep_one(src)
52640
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52359
diff changeset
714 except OSError as err:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
715 if err.errno not in (errno.ENOENT, errno.EPERM):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
716 raise
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
717 return None
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
718
52013
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
719 @contextlib.contextmanager
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
720 def open(self, src):
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
721 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
722 entry = self._volatile_fps.get(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
723 if entry is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
724 _size, fp = entry
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
725 fp.seek(0)
52013
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
726 yield fp
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
727 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
728 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
729 actual_path = src
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
730 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
731 actual_path = self._copies.get(src, src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
732 with open(actual_path, 'rb') as fp:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
733 yield fp
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
734
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
735
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
736 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
737 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
738 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
739 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
740 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
741 }
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
742 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
743 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
744 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
745
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
746 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
747
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
748
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
749 def _emit2(repo, entries):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
750 """actually emit the stream bundle"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
751 vfsmap = _makemap(repo)
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
752 # we keep repo.vfs out of the on purpose, ther are too many danger there
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
753 # (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
754 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
755 # this assert is duplicated (from _makemap) as author might think this is
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
756 # fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
757 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
758 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
759 'repo.vfs must not be added to vfsmap for security reasons'
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
760 )
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
761
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
762 # translate the vfs one
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
763 entries = [(vfs_key, vfsmap[vfs_key], e) for (vfs_key, e) in entries]
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
764 _test_sync_point_walk_1_2(repo)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
765
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
766 max_linkrev = len(repo)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
767 file_count = totalfilesize = 0
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
768 with VolatileManager() as volatiles:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
769 # make sure we preserve volatile files
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
770 with util.nogc():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
771 # record the expected size of every file
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
772 for k, vfs, e in entries:
52357
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52356
diff changeset
773 e.preserve_volatiles(vfs, volatiles)
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
774 for f in e.files():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
775 file_count += 1
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
776 totalfilesize += f.file_size(vfs)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
777
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
778 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
779 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
780 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
781 progress.update(0)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
782 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
783 # the first yield release the lock on the repository
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
784 yield file_count, totalfilesize
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
785 totalbytecount = 0
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
786
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
787 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
788 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
789 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
790 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
791 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
792 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
793 preserve_file_count=True,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
794 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
795 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
796 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
797 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
798 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
799 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
800 bytecount = 0
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
801 for chunk in stream:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
802 bytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
803 totalbytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
804 progress.update(totalbytecount)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
805 yield chunk
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
806 if bytecount != size:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
807 # Would most likely be caused by a race due to `hg
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
808 # strip` or a revlog split
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
809 msg = _(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
810 b'clone could only read %d bytes from %s, but '
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
811 b'expected %d bytes'
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
812 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
813 raise error.Abort(msg % (bytecount, name, size))
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
814
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
815
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
816 def _emit3(repo, entries) -> Iterator[bytes | None]:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
817 """actually emit the stream bundle (v3)"""
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
818 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
819 # we keep repo.vfs out of the map on purpose, ther are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
820 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
821 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
822 # this assert is duplicated (from _makemap) as authors might think this is
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
823 # fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
824 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
825 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
826 'repo.vfs must not be added to vfsmap for security reasons'
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
827 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
828
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
829 # translate the vfs once
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
830 # we only turn this into a list for the `_test_sync`, this is not ideal
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
831 base_entries = list(entries)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
832 _test_sync_point_walk_1_2(repo)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
833 entries = []
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
834 with VolatileManager() as volatiles:
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51859
diff changeset
835 # make sure we preserve volatile files
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
836 for vfs_key, e in base_entries:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
837 vfs = vfsmap[vfs_key]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
838 any_files = True
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
839 if e.maybe_volatile:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
840 any_files = False
52357
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52356
diff changeset
841 e.preserve_volatiles(vfs, volatiles)
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
842 for f in e.files():
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
843 if f.is_volatile:
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
844 # record the expected size under lock
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
845 f.file_size(vfs)
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
846 any_files = True
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
847 if any_files:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
848 entries.append((vfs_key, vfsmap[vfs_key], e))
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
849
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
850 total_entry_count = len(entries)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
851
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
852 max_linkrev = len(repo)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
853 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
854 _(b'bundle'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
855 total=total_entry_count,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
856 unit=_(b'entry'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
857 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
858 progress.update(0)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
859 # the first yield release the lock on the repository
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
860 yield None
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
861 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
862 yield util.uvarintencode(total_entry_count)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
863
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
864 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
865 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
866 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
867 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
868 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
869 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
870 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
871 yield util.uvarintencode(len(entry_streams))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
872 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
873 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
874 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
875 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
876 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
877 yield from stream
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
878 progress.increment()
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
879
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
880
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
881 def _test_sync_point_walk_1_2(repo):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
882 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
883
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
884 Triggered after gather entry, but before starting to process/preserve them
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
885 under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
886
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
887 (on v1 is triggered before the actual walk start)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
888 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
889
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
890
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
891 def _test_sync_point_walk_3(repo):
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
892 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
893
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
894 Triggered right before releasing the lock, but after computing what need
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
895 needed to compute under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
896 """
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
897
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
898
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
899 def _test_sync_point_walk_4(repo):
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
900 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
901
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
902 Triggered right after releasing the lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
903 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
904
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
905
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
906 # not really a StoreEntry, but close enough
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
907 class CacheEntry(store.SimpleStoreEntry):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
908 """Represent an entry for Cache files
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
909
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
910 It has special logic to preserve cache file early and accept optional
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
911 presence.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
912
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
913
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
914 (Yes... this is not really a StoreEntry, but close enough. We could have a
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
915 BaseEntry base class, bbut the store one would be identical)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
916 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
917
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
918 def __init__(self, entry_path) -> None:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
919 super().__init__(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
920 entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
921 # we will directly deal with that in `setup_cache_file`
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
922 is_volatile=True,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
923 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
924
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
925 def preserve_volatiles(self, vfs, volatiles) -> None:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
926 self._file_size = volatiles.try_keep(vfs.join(self._entry_path))
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
927 if self._file_size is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
928 self._files = []
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
929 else:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
930 assert self._is_volatile
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
931 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
932 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
933 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
934 file_size=self._file_size,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
935 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
936 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
937 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
938
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
939 def files(self) -> list[store.StoreFile]:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
940 if self._files is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
941 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
942 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
943 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
944 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
945 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
946 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
947 return super().files()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
948
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
949
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
950 class CacheFile(store.StoreFile):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
951 # inform the "copy/hardlink" version that this file might be missing
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
952 # without consequences.
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
953 optional: bool = True
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
954
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
955
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
956 def _entries_walk(repo, includes, excludes, includeobsmarkers: bool):
50512
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
957 """emit a seris of files information useful to clone a repo
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
958
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
959 return (vfs-key, entry) iterator
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
960
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
961 Where `entry` is StoreEntry. (used even for cache entries)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
962 """
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
963 assert repo._currentlock(repo._lockref) is not None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
964
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
965 matcher = None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
966 if includes or excludes:
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
967 matcher = narrowspec.match(repo.root, includes, excludes)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
968
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
969 phase = not repo.publishing()
51547
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
970 # Python is getting crazy at all the small container we creates, disabling
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
971 # the gc while we do so helps performance a lot.
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
972 with util.nogc():
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
973 entries = _walkstreamfiles(
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
974 repo,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
975 matcher,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
976 phase=phase,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
977 obsolescence=includeobsmarkers,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
978 )
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
979 for entry in entries:
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
980 yield (_srcstore, entry)
50512
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
981
51547
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
982 for name in cacheutil.cachetocopy(repo):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
983 if repo.cachevfs.exists(name):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
984 # not really a StoreEntry, but close enough
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
985 yield (_srccache, CacheEntry(entry_path=name))
50513
43ed1f12b00a stream-clone: yield cache entry in `_entries_walk` too
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50512
diff changeset
986
50512
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
987
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
988 def generatev2(repo, includes, excludes, includeobsmarkers: bool):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
989 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
990
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
991 the data stream consists the following entries:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
992 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
993 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
994 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
995 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
996 5) N bytes containing the file data
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
997
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
998 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
999 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1000
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1001 with repo.lock():
47444
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1002 repo.ui.debug(b'scanning\n')
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
1003
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1004 entries = _entries_walk(
47444
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1005 repo,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1006 includes=includes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1007 excludes=excludes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1008 includeobsmarkers=includeobsmarkers,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1009 )
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1010 chunks = _emit2(repo, entries)
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
1011 first = next(chunks)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1012 file_count, total_file_size = first
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1013 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1014 _test_sync_point_walk_4(repo)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1015
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1016 return file_count, total_file_size, chunks
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1017
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1018
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1019 def generatev3(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1020 repo, includes, excludes, includeobsmarkers: bool
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1021 ) -> Iterator[bytes | None]:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1022 """Emit content for version 3 of a streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1023
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1024 the data stream consists the following:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1025 1) A varint E containing the number of entries (can be 0), then E entries follow
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1026 2) For each entry:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1027 2.1) The number of files in this entry (can be 0, but typically 1 or 2)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1028 2.2) For each file:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1029 2.2.1) A char representing the file destination (eg: store or cache)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1030 2.2.2) A varint N containing the length of the filename
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1031 2.2.3) A varint M containing the length of file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1032 2.2.4) N bytes containing the filename (the internal, store-agnostic form)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1033 2.2.5) M bytes containing the file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1034
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1035 Returns the data iterator.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1036
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1037 XXX This format is experimental and subject to change. Here is a
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1038 XXX non-exhaustive list of things this format could do or change:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1039
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1040 - making it easier to write files in parallel
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1041 - holding the lock for a shorter time
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1042 - improving progress information
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1043 - ways to adjust the number of expected entries/files ?
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1044 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1045
51548
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1046 # Python is getting crazy at all the small container we creates while
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1047 # considering the files to preserve, disabling the gc while we do so helps
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1048 # performance a lot.
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1049 with repo.lock(), util.nogc():
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1050 repo.ui.debug(b'scanning\n')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1051
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1052 entries = _entries_walk(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1053 repo,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1054 includes=includes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1055 excludes=excludes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1056 includeobsmarkers=includeobsmarkers,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1057 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1058 chunks = _emit3(repo, list(entries))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1059 first = next(chunks)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1060 assert first is None
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1061 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1062 _test_sync_point_walk_4(repo)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1063
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1064 return chunks
50521
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50520
diff changeset
1065
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50520
diff changeset
1066
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1067 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1068 def nested(*ctxs):
39757
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1069 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1070 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1071 with this:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1072 if rest:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1073 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1074 yield
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1075 else:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1076 yield
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1077
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1078
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1079 class V2Report:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1080 """a small class to track the data we saw within the stream"""
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1081
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1082 def __init__(self):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1083 self.byte_count = 0
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1084
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1085
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1086 def consumev2(repo, fp, filecount: int, filesize: int) -> None:
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1087 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1088
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1089 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1090 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1091 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1092 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1093 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1094 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1095 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1096 )
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1097 progress = repo.ui.makeprogress(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1098 _(b'clone'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1099 total=filesize,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1100 unit=_(b'bytes'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1101 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1102 start = util.timer()
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1103 report = V2Report()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1104
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1105 vfsmap = _makemap(repo)
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1106 # we keep repo.vfs out of the on purpose, ther are too many danger
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1107 # there (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1108 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1109 # this assert is duplicated (from _makemap) as author might think this
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1110 # is fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1111 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1112 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
1113 'repo.vfs must not be added to vfsmap for security reasons'
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1114 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1115
52913
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
1116 cpu_profile = scmutil.get_resource_profile(repo.ui, b'cpu')
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1117 threaded = repo.ui.configbool(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1118 b"worker", b"parallel-stream-bundle-processing"
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1119 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1120 num_writer = repo.ui.configint(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1121 b"worker",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1122 b"parallel-stream-bundle-processing.num-writer",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1123 )
52913
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
1124 if num_writer <= 0:
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
1125 num_writer = DEFAULT_NUM_WRITER[cpu_profile]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1126 with repo.transaction(b'clone'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1127 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1128 with nested(*ctxs):
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1129 if threaded:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1130 fc = _ThreadSafeFileChunker
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1131 else:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1132 fc = _FileChunker
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1133 files = _v2_parse_files(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1134 repo,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1135 fp,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1136 filecount,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1137 progress,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1138 report,
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1139 file_chunker=fc,
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1140 )
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1141 if not threaded:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1142 _write_files(vfsmap, files)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1143 else:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1144 info_queue = _FileInfoQueue(files)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1145
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1146 workers = []
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1147 try:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1148 for __ in range(num_writer):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1149 w = threading.Thread(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1150 target=_write_files,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1151 args=(vfsmap, info_queue),
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1152 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1153 workers.append(w)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1154 w.start()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1155 info_queue.fill()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1156 finally:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1157 # shut down all the workers
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1158 info_queue.close(len(workers))
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1159 for w in workers:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1160 w.join()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1161
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1162 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1163 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1164 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1165
38373
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38360
diff changeset
1166 progress.complete()
52703
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1167 # acknowledge the end of the bundle2 part, this help aligning
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1168 # sequential and parallel behavior.
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1169 remains = fp.read(1)
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1170 assert not remains
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1171 _report_transferred(repo, start, filecount, report.byte_count)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1172
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1173
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1174 # iterator of chunk of bytes that constitute a file content.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1175 FileChunksT = Iterator[bytes]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1176 # Contains the information necessary to write stream file on disk
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1177 FileInfoT = Tuple[
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1178 bytes, # vfs key
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1179 bytes, # file name (non-vfs-encoded)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1180 FileChunksT, # content
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1181 ]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1182
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1183
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1184 class _FileInfoQueue:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1185 """A thread-safe queue to passer parsed file information to the writers"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1186
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1187 def __init__(self, info: Iterable[FileInfoT]):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1188 self._info = info
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1189 self._q = queue.Queue()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1190
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1191 def fill(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1192 """iterate over the parsed information to file the queue
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1193
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1194 This is meant to be call from the thread parsing the stream information.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1195 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1196 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1197 for i in self._info:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1198 q.put(i)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1199
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1200 def close(self, number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1201 """signal all the workers that we no longer have any file info coming
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1202
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1203 Called from the thread parsing the stream information (and/or the main
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1204 thread if different).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1205 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1206 for __ in range(number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1207 self._q.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1208
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1209 def __iter__(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1210 """iterate over the available file info
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1211
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1212 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1213 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1214 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1215 while (i := q.get()) is not None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1216 yield i
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1217
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1218
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1219 class _FileChunker:
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1220 """yield the chunk that constitute a file
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1221
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1222 This class exists as the counterpart of the threaded version and
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1223 would not be very useful on its own.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1224 """
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1225
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1226 def __init__(
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1227 self,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1228 fp: bundle2mod.unbundlepart,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1229 data_len: int,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1230 progress: scmutil.progress,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1231 report: V2Report,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1232 ):
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1233 self.report = report
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1234 self.progress = progress
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1235 self._chunks = util.filechunkiter(fp, limit=data_len)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1236
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1237 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1238 """Do nothing in non-threading context"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1239
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1240 def __iter__(self) -> FileChunksT:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1241 for chunk in self._chunks:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1242 self.report.byte_count += len(chunk)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1243 self.progress.increment(step=len(chunk))
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1244 yield chunk
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1245
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1246
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1247 class _ThreadSafeFileChunker(_FileChunker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1248 """yield the chunk that constitute a file
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1249
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1250 Make sure you call the "fill" function in the main thread to read the
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1251 right data at the right time.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1252 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1253
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1254 def __init__(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1255 self,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1256 fp: bundle2mod.unbundlepart,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1257 data_len: int,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1258 progress: scmutil.progress,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1259 report: V2Report,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1260 ):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1261 super().__init__(fp, data_len, progress, report)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1262 self._queue = queue.Queue()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1263
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1264 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1265 """fill the file chunker queue with data read from the stream
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1266
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1267 This is meant to be called from the thread parsing information (and
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1268 consuming the stream data).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1269 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1270 try:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1271 for chunk in super().__iter__():
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1272 self._queue.put(chunk)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1273 finally:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1274 self._queue.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1275
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1276 def __iter__(self) -> FileChunksT:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1277 """Iterate over all the file chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1278
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1279 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1280 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1281 while (chunk := self._queue.get()) is not None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1282 yield chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1283
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1284
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1285 def _v2_parse_files(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1286 repo,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1287 fp: bundle2mod.unbundlepart,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1288 file_count: int,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1289 progress: scmutil.progress,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1290 report: V2Report,
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1291 file_chunker: Type[_FileChunker] = _FileChunker,
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1292 ) -> Iterator[FileInfoT]:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1293 """do the "stream-parsing" part of stream v2
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1294
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1295 The parsed information are yield result for consumption by the "writer"
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1296 """
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1297 progress.update(0)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1298 for i in range(file_count):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1299 src = util.readexactly(fp, 1)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1300 namelen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1301 datalen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1302
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1303 name = util.readexactly(fp, namelen)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1304
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1305 if repo.ui.debugflag:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1306 repo.ui.debug(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1307 b'adding [%s] %s (%s)\n' % (src, name, util.bytecount(datalen))
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1308 )
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1309 chunks = file_chunker(fp, datalen, progress, report)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1310 yield (src, name, iter(chunks))
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1311 # make sure we read all the chunk before moving to the next file
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1312 chunks.fill()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1313
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1314
52910
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1315 def _write_files(vfsmap, info: Iterable[FileInfoT]):
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1316 """write files from parsed data"""
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1317 for src, name, data in info:
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1318 vfs = vfsmap[src]
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1319 with vfs(name, b'w') as ofp:
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1320 for chunk in data:
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1321 ofp.write(chunk)
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1322
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1323
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1324 def consumev3(repo, fp) -> None:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1325 """Apply the contents from a version 3 streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1326
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1327 Data is read from an object that only needs to provide a ``read(size)``
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1328 method.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1329 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1330 with repo.lock():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1331 start = util.timer()
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1332
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1333 entrycount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1334 repo.ui.status(_(b'%d entries to transfer\n') % (entrycount))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1335
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1336 progress = repo.ui.makeprogress(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1337 _(b'clone'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1338 total=entrycount,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1339 unit=_(b'entries'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1340 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1341 progress.update(0)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1342 bytes_transferred = 0
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1343
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1344 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1345 # we keep repo.vfs out of the on purpose, there are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1346 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1347 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1348 # this assert is duplicated (from _makemap) as authors might think this
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1349 # is fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1350 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1351 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
1352 'repo.vfs must not be added to vfsmap for security reasons'
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1353 )
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
1354 total_file_count = 0
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1355 with repo.transaction(b'clone'):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1356 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1357 with nested(*ctxs):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1358 for i in range(entrycount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1359 filecount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1360 if filecount == 0:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1361 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1362 repo.ui.debug(b'entry with no files [%d]\n' % (i))
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
1363 total_file_count += filecount
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1364 for i in range(filecount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1365 src = util.readexactly(fp, 1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1366 vfs = vfsmap[src]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1367 namelen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1368 datalen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1369
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1370 name = util.readexactly(fp, namelen)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1371
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1372 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1373 msg = b'adding [%s] %s (%s)\n'
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1374 msg %= (src, name, util.bytecount(datalen))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1375 repo.ui.debug(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1376 bytes_transferred += datalen
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1377
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1378 with vfs(name, b'w') as ofp:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1379 for chunk in util.filechunkiter(fp, limit=datalen):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1380 ofp.write(chunk)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1381 progress.increment(step=1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1382
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1383 # force @filecache properties to be reloaded from
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1384 # streamclone-ed file at next access
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1385 repo.invalidate(clearfilecache=True)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1386
52701
6feb3b3029b5 stream: consistently close progress before reporting time
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52700
diff changeset
1387 progress.complete()
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
1388 _report_transferred(repo, start, total_file_count, bytes_transferred)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1389
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1390
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1391 def applybundlev2(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1392 repo, fp, filecount: int, filesize: int, requirements: Iterable[bytes]
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1393 ) -> None:
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
1394 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
1395
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1396 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1397 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1398 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
1399 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1400 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1401 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1402
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1403 consumev2(repo, fp, filecount, filesize)
35804
2d3e486d09d0 streamclone: move requirement update into consumev2
Boris Feld <boris.feld@octobus.net>
parents: 35803
diff changeset
1404
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
1405 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
1406 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
1407 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1408 )
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
1409 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1410 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1411 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
1412 scmutil.writereporequirements(repo)
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
1413 nodemap.post_stream_cleanup(repo)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1414
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1415
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1416 def applybundlev3(repo, fp, requirements: Iterable[bytes]) -> None:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1417 from . import localrepo
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1418
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1419 missingreqs = [r for r in requirements if r not in repo.supported]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1420 if missingreqs:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1421 msg = _(b'unable to apply stream clone: unsupported format: %s')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1422 msg %= b', '.join(sorted(missingreqs))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1423 raise error.Abort(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1424
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1425 consumev3(repo, fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1426
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1427 repo.requirements = new_stream_clone_requirements(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1428 repo.requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1429 requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1430 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1431 repo.svfs.options = localrepo.resolvestorevfsoptions(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1432 repo.ui, repo.requirements, repo.features
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1433 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1434 scmutil.writereporequirements(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1435 nodemap.post_stream_cleanup(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1436
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1437
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1438 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress) -> bool:
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1439 hardlink = [True]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1440
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1441 def copy_used():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1442 hardlink[0] = False
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1443 progress.topic = _(b'copying')
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1444
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1445 for k, path, optional in entries:
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1446 src_vfs = src_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1447 dst_vfs = dst_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1448 src_path = src_vfs.join(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1449 dst_path = dst_vfs.join(path)
47871
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1450 # We cannot use dirname and makedirs of dst_vfs here because the store
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1451 # encoding confuses them. See issue 6581 for details.
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1452 dirname = os.path.dirname(dst_path)
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1453 if not os.path.exists(dirname):
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1454 util.makedirs(dirname)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1455 dst_vfs.register_file(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1456 # XXX we could use the #nb_bytes argument.
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1457 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1458 util.copyfile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1459 src_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1460 dst_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1461 hardlink=hardlink[0],
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1462 no_hardlink_cb=copy_used,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1463 check_fs_hardlink=False,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1464 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1465 except FileNotFoundError:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1466 if not optional:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1467 raise
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1468 progress.increment()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1469 return hardlink[0]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1470
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1471
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1472 def local_copy(src_repo, dest_repo) -> None:
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1473 """copy all content from one local repository to another
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1474
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1475 This is useful for local clone"""
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1476 src_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1477 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1478 for r in src_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1479 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1480 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1481 dest_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1482 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1483 for r in dest_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1484 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1485 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1486 assert src_store_requirements == dest_store_requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1487
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1488 with dest_repo.lock():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1489 with src_repo.lock():
47448
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1490 # bookmark is not integrated to the streaming as it might use the
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1491 # `repo.vfs` and they are too many sentitive data accessible
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1492 # through `repo.vfs` to expose it to streaming clone.
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1493 src_book_vfs = bookmarks.bookmarksvfs(src_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1494 srcbookmarks = src_book_vfs.join(b'bookmarks')
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1495 bm_count = 0
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1496 if os.path.exists(srcbookmarks):
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1497 bm_count = 1
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1498
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1499 entries = _entries_walk(
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1500 src_repo,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1501 includes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1502 excludes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1503 includeobsmarkers=True,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1504 )
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1505 entries = list(entries)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1506 src_vfs_map = _makemap(src_repo)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1507 dest_vfs_map = _makemap(dest_repo)
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1508 total_files = sum(len(e[1].files()) for e in entries) + bm_count
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1509 progress = src_repo.ui.makeprogress(
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1510 topic=_(b'linking'),
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1511 total=total_files,
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1512 unit=_(b'files'),
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1513 )
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1514 # copy files
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1515 #
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1516 # We could copy the full file while the source repository is locked
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1517 # and the other one without the lock. However, in the linking case,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1518 # this would also requires checks that nobody is appending any data
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1519 # to the files while we do the clone, so this is not done yet. We
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1520 # could do this blindly when copying files.
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1521 files = [
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1522 (vfs_key, f.unencoded_path, f.optional)
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1523 for vfs_key, e in entries
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1524 for f in e.files()
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1525 ]
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1526 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1527
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1528 # copy bookmarks over
47448
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1529 if bm_count:
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1530 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1531 dstbookmarks = dst_book_vfs.join(b'bookmarks')
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1532 util.copyfile(srcbookmarks, dstbookmarks)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1533 progress.complete()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1534 if hardlink:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1535 msg = b'linked %d files\n'
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1536 else:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1537 msg = b'copied %d files\n'
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1538 src_repo.ui.debug(msg % total_files)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1539
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1540 with dest_repo.transaction(b"localclone") as tr:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1541 dest_repo.store.write(tr)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1542
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1543 # clean up transaction file as they do not make sense
50288
d89eecf9605e undo-files: no longer pass the `repo` to `cleanup_undo_files`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50286
diff changeset
1544 transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)