Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/streamclone.py @ 52713:f5471af96a52

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
typing: add trivial type annotations to `mercurial/streamclone.py` These are the easy/obvious/documented ones. We'll leave the harder ones for later.
author Matt Harbison <matt_harbison@yahoo.com>
date Mon, 13 Jan 2025 00:36:25 -0500
parents 279e217d6041
children 10e7adbffa8c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
51901
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51762
diff changeset
8 from __future__ import annotations
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
10 import contextlib
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
11 import errno
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
12 import os
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
13 import struct
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
14
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
15 from typing import Iterable, Iterator, Optional, Set
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
16
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
17 from .i18n import _
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
18 from .interfaces import repository
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
19 from . import (
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
20 bookmarks,
50557
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
21 bundle2 as bundle2mod,
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
22 cacheutil,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
23 error,
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
24 narrowspec,
32764
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
25 phases,
38197
aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path
Yuya Nishihara <yuya@tcha.org>
parents: 37638
diff changeset
26 pycompat,
46665
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
27 requirements as requirementsmod,
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
28 scmutil,
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
29 store,
50286
3d0b5760851c undo-files: move the undo cleanup code in the transaction module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50279
diff changeset
30 transaction,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
31 util,
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
32 )
48707
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
33 from .revlogutils import (
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
34 nodemap,
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
35 )
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
36
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
37
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
38 def new_stream_clone_requirements(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
39 default_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
40 streamed_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
41 ) -> Set[bytes]:
48618
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
42 """determine the final set of requirement for a new stream clone
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
43
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
44 this method combine the "default" requirements that a new repository would
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
45 use with the constaint we get from the stream clone content. We keep local
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
46 configuration choice when possible.
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
47 """
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
48 requirements = set(default_requirements)
48623
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48622
diff changeset
49 requirements -= requirementsmod.STREAM_FIXED_REQUIREMENTS
48618
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
50 requirements.update(streamed_requirements)
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
51 return requirements
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
52
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
53
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
54 def streamed_requirements(repo) -> Set[bytes]:
48619
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
55 """the set of requirement the new clone will have to support
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
56
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
57 This is used for advertising the stream options and to generate the actual
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
58 stream content."""
48623
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48622
diff changeset
59 requiredformats = (
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48622
diff changeset
60 repo.requirements & requirementsmod.STREAM_FIXED_REQUIREMENTS
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48622
diff changeset
61 )
48619
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
62 return requiredformats
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
63
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
64
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
65 def canperformstreamclone(pullop, bundle2: bool = False):
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
66 """Whether it is possible to perform a streaming clone as part of pull.
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
67
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
68 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
69 bundle2 and only through bundle2.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
70
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
71 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
72 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
73 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
74 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
75 """
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
76 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
77 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
78
50556
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
79 # should we consider streaming clone at all ?
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
80 streamrequested = pullop.streamclonerequested
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
81 # If we don't have a preference, let the server decide for us. This
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
82 # likely only comes into play in LANs.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
83 if streamrequested is None:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
84 # The server can advertise whether to prefer streaming clone.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
85 streamrequested = remote.capable(b'stream-preferred')
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
86 if not streamrequested:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
87 return False, None
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50555
diff changeset
88
50554
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50550
diff changeset
89 # Streaming clone only works on an empty destination repository
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50550
diff changeset
90 if len(repo):
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50550
diff changeset
91 return False, None
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50550
diff changeset
92
50555
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50554
diff changeset
93 # Streaming clone only works if all data is being requested.
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50554
diff changeset
94 if pullop.heads:
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50554
diff changeset
95 return False, None
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50554
diff changeset
96
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
97 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
98 if pullop.canusebundle2:
50557
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
99 local_caps = bundle2mod.getrepocaps(repo, role=b'client')
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
100 local_supported = set(local_caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
101 remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50556
diff changeset
102 bundle2supported = bool(local_supported & remote_supported)
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
103 # else
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
104 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
105 # the versions we support. Fall back and possibly allow legacy.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
106
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
107 # Ensures legacy code path uses available bundle2.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
108 if bundle2supported and not bundle2:
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
109 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
110 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
111 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
112 return False, None
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
113
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
114 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
115 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
116 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
117 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
118 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
119 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
120 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
121 requirements = set()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
122 if remote.capable(b'stream'):
46665
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
123 requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
124 else:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
125 streamreqs = remote.capable(b'streamreqs')
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
126 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
127 if not streamreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
128 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
129 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
130 b'warning: stream clone requested but server has them '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
131 b'disabled\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
132 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
133 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
134 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
135
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
136 streamreqs = set(streamreqs.split(b','))
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
137 # Server requires something we don't support. Bail.
48359
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
138 missingreqs = streamreqs - repo.supported
32299
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30995
diff changeset
139 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30995
diff changeset
140 pullop.repo.ui.warn(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
141 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
142 b'warning: stream clone requested but client is missing '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
143 b'requirements: %s\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
144 )
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
145 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
146 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
147 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
148 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
149 b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
150 b'for more information)\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
151 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
152 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
153 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
154 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
155
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
156 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
157
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
158
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
159 def maybeperformlegacystreamclone(pullop) -> None:
26462
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
160 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
161
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
162 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
163 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
164
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
165 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
166 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
167 """
39716
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
168 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
169
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
170 supported, requirements = canperformstreamclone(pullop)
26458
362793295640 streamclone: refactor maybeperformstreamclone to take a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26447
diff changeset
171
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
172 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
173 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
174
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
175 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
176 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
177
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
178 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
179 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
180 rbranchmap = None
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
181 if remote.capable(b'branchmap'):
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
182 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
183 rbranchmap = e.callcommand(b'branchmap', {}).result()
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
184
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
185 repo.ui.status(_(b'streaming all changes\n'))
26470
4b5647d9ee13 streamclone: move "streaming all changes" message location
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26469
diff changeset
186
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
187 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
188 fp = e.callcommand(b'stream_out', {}).result()
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
189
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
190 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
191 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
192 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
193 # doesn't matter.
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
194 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
195 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
196 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
197 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
198 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
199 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
200 )
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
201 if resp == 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
202 raise error.Abort(_(b'operation forbidden by server'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
203 elif resp == 2:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
204 raise error.Abort(_(b'locking the remote repository failed'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
205 elif resp != 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
206 raise error.Abort(_(b'the server sent an unknown error code'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
207
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
208 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
209 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
210 filecount, bytecount = map(int, l.split(b' ', 1))
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
211 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
212 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
213 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
214 )
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
215
27850
49cfddbf54ba with: use context manager in maybeperformlegacystreamclone
Bryan O'Sullivan <bryano@fb.com>
parents: 27845
diff changeset
216 with repo.lock():
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
217 consumev1(repo, fp, filecount, bytecount)
48618
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
218 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
219 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
220 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
221 )
39716
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
222 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
223 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
224 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
225 scmutil.writereporequirements(repo)
48707
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
226 nodemap.post_stream_cleanup(repo)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
227
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
228 if rbranchmap:
41626
328ca3b9e545 branchmap: encapsulate cache updating in the map itself
Martijn Pieters <mj@octobus.net>
parents: 40544
diff changeset
229 repo._branchcaches.replace(repo, rbranchmap)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
230
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
231 repo.invalidate()
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
232
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
233
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
234 def allowservergeneration(repo) -> bool:
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
235 """Whether streaming clones are allowed from the server."""
40029
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39855
diff changeset
236 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39855
diff changeset
237 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39855
diff changeset
238
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
239 if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
32764
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
240 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
241
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
242 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
243 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
244 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
245 if secret:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
246 return repo.ui.configbool(b'server', b'uncompressedallowsecret')
32764
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
247
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32299
diff changeset
248 return True
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
249
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
250
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
251 # This is it's own function so extensions can override it.
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
252 def _walkstreamfiles(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
253 repo, matcher=None, phase: bool = False, obsolescence: bool = False
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
254 ):
50548
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50546
diff changeset
255 return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
256
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
257
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
258 def generatev1(repo) -> tuple[int, int, Iterator[bytes]]:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
259 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
260
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
261 This returns a 3-tuple of (file count, byte size, data iterator).
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
262
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
263 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
264 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
265 delimited by a null byte.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
266
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
267 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
268 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
269
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
270 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
271 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
272 for adding it.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
273
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
274 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
275 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
276 """
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
277 entries = []
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
278 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
279 # Get consistent snapshot of repo, lock during scan.
27845
7417e1c10253 with: use context manager in streamclone generatev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27794
diff changeset
280 with repo.lock():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
281 repo.ui.debug(b'scanning\n')
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
282 _test_sync_point_walk_1_2(repo)
50505
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50288
diff changeset
283 for entry in _walkstreamfiles(repo):
50506
9fdc28e21b68 store: introduce a EntryFile object to actually access file info
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50505
diff changeset
284 for f in entry.files():
50511
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50506
diff changeset
285 file_size = f.file_size(repo.store.vfs)
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50506
diff changeset
286 if file_size:
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50506
diff changeset
287 entries.append((f.unencoded_path, file_size))
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50506
diff changeset
288 total_bytes += file_size
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
289 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
290 _test_sync_point_walk_4(repo)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
291
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
292 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
293 b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
294 )
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
295
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
296 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
297 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
298
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
299 def emitrevlogdata() -> Iterator[bytes]:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
300 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
301 if debugflag:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
302 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
303 # partially encode name over the wire for backwards compat
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
304 yield b'%s\0%d\n' % (store.encodedir(name), size)
33411
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
305 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
306 # trusted by the local repo) and expensive
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
307 with svfs(name, b'rb', auditpath=False) as fp:
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
308 if size <= 65536:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
309 yield fp.read(size)
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
310 else:
52669
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
311 yield from util.filechunkiter(fp, limit=size)
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
312
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
313 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
314
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
315
52709
279e217d6041 typing: lock in the new type annotations detected with the pyupgrade changes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52669
diff changeset
316 def generatev1wireproto(repo) -> Iterator[bytes]:
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
317 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
318
35495
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
319 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
320 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
321 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
322
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
323 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
324 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
325 a permissions error for the server process).
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
326 """
35495
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
327 if not allowservergeneration(repo):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
328 yield b'1\n'
35495
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
329 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
330
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
331 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
332 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
333 except error.LockError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
334 yield b'2\n'
35495
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
335 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
336
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
337 # Indicates successful response.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
338 yield b'0\n'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
339 yield b'%d %d\n' % (filecount, bytecount)
52669
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52665
diff changeset
340 yield from it
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
341
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
342
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
343 def generatebundlev1(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
344 repo, compression: bytes = b'UN'
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
345 ) -> tuple[Set[bytes], Iterator[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
346 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
347
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
348 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
349 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
350
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
351 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
352 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
353
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
354 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
355 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
356
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
357 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
358 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
359 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
360 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
361
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
362 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
363 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
364
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
365 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
366 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
367 if compression != b'UN':
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
368 raise ValueError(b'we do not support the compression argument yet')
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
369
48619
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48618
diff changeset
370 requirements = streamed_requirements(repo)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
371 requires = b','.join(sorted(requirements))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
372
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
373 def gen() -> Iterator[bytes]:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
374 yield b'HGS1'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
375 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
376
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
377 filecount, bytecount, it = generatev1(repo)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
378 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
379 _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
380 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
381
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
382 yield struct.pack(b'>QQ', filecount, bytecount)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
383 yield struct.pack(b'>H', len(requires) + 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
384 yield requires + b'\0'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
385
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
386 # This is where we'll add compression in the future.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
387 assert compression == b'UN'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
388
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
389 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
390 _(b'bundle'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
391 )
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
392 progress.update(0)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
393
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
394 for chunk in it:
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
395 progress.increment(step=len(chunk))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
396 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
397
38379
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38366
diff changeset
398 progress.complete()
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
399
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
400 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
401
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
402
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
403 def consumev1(repo, fp, filecount: int, bytecount: int) -> None:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
404 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
405
30342
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29923
diff changeset
406 This takes the output from "stream_out" and applies it to the specified
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
407 repository.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
408
30342
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29923
diff changeset
409 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29923
diff changeset
410 handled by this function.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
411 """
27859
f55a5ace8e69 with: use context manager in streamclone consumev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27850
diff changeset
412 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
413 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
414 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
415 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
416 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
417 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
418 _(b'clone'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
419 )
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
420 progress.update(0)
30995
22fbca1d11ed mercurial: switch to util.timer for all interval timings
Simon Farnsworth <simonfar@fb.com>
parents: 30342
diff changeset
421 start = util.timer()
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
422
29923
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
423 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
424 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
425 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
426 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
427 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
428 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
429 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
430 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
431 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
432 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
433 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
434 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
435 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
436
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
437 with repo.transaction(b'clone'):
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
438 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
49292
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 49222
diff changeset
439 for i in range(filecount):
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
440 # XXX doesn't support '\n' or '\r' in filenames
50951
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50706
diff changeset
441 if hasattr(fp, 'readline'):
50675
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
442 l = fp.readline()
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
443 else:
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
444 # inline clonebundles use a chunkbuffer, so no readline
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
445 # --> this should be small anyway, the first line
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
446 # only contains the size of the bundle
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
447 l_buf = []
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
448 while not (l_buf and l_buf[-1] == b'\n'):
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
449 l_buf.append(fp.read(1))
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50670
diff changeset
450 l = b''.join(l_buf)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
451 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
452 name, size = l.split(b'\0', 1)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
453 size = int(size)
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
454 except (ValueError, TypeError):
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
455 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
456 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
457 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
458 if repo.ui.debugflag:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
459 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
460 b'adding %s (%s)\n' % (name, util.bytecount(size))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
461 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
462 # for backwards compat, name was partially encoded
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
463 path = store.decodedir(name)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
464 with repo.svfs(path, b'w', backgroundclose=True) as ofp:
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
465 for chunk in util.filechunkiter(fp, limit=size):
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
466 progress.increment(step=len(chunk))
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
467 ofp.write(chunk)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
468
29923
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
469 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
470 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29921
diff changeset
471 repo.invalidate(clearfilecache=True)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
472
30995
22fbca1d11ed mercurial: switch to util.timer for all interval timings
Simon Farnsworth <simonfar@fb.com>
parents: 30342
diff changeset
473 elapsed = util.timer() - start
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
474 if elapsed <= 0:
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
475 elapsed = 0.001
38379
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38366
diff changeset
476 progress.complete()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
477 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
478 _(b'transferred %s in %.1f seconds (%s/sec)\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
479 % (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
480 util.bytecount(bytecount),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
481 elapsed,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
482 util.bytecount(bytecount / elapsed),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
483 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
484 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
485
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
486
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
487 def readbundle1header(fp) -> tuple[int, int, Set[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
488 compression = fp.read(2)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
489 if compression != b'UN':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
490 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
491 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
492 b'only uncompressed stream clone bundles are '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
493 b'supported; got %s'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
494 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
495 % compression
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
496 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
497
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
498 filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
499 requireslen = struct.unpack(b'>H', fp.read(2))[0]
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
500 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
501
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
502 if not requires.endswith(b'\0'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
503 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
504 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
505 b'malformed stream clone bundle: '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
506 b'requirements not properly encoded'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
507 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
508 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
509
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
510 requirements = set(requires.rstrip(b'\0').split(b','))
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
511
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
512 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
513
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
514
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
515 def applybundlev1(repo, fp) -> None:
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
516 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
517
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
518 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
519 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
520 """
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
521 if len(repo):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
522 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
523 _(b'cannot apply stream clone bundle on non-empty repo')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
524 )
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
525
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
526 filecount, bytecount, requirements = readbundle1header(fp)
48359
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
527 missingreqs = requirements - repo.supported
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
528 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
529 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
530 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
531 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
532 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
533
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
534 consumev1(repo, fp, filecount, bytecount)
48707
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
535 nodemap.post_stream_cleanup(repo)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
536
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
537
49037
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48966
diff changeset
538 class streamcloneapplier:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
539 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
540
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
541 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
542 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
543 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
544
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
545 def __init__(self, fh) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
546 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
547
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
548 def apply(self, repo) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
549 return applybundlev1(repo, self._fh)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
550
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
551
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
552 # type of file to stream
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
553 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
554 _filefull = 1 # full snapshot file
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
555
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
556 # Source of the file
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
557 _srcstore = b's' # store (svfs)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
558 _srccache = b'c' # cache (cache)
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
559
51762
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51759
diff changeset
560
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
561 # This is it's own function so extensions can override it.
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
562 def _walkstreamfullstorefiles(repo) -> list[bytes]:
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
563 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
564 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
565 if not repo.publishing():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
566 fnames.append(b'phaseroots')
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
567 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
568
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
569
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
570 def _filterfull(entry, copy, vfsmap):
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
571 """actually copy the snapshot files"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
572 src, name, ftype, data = entry
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
573 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
574 return entry
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
575 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
576
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
577
52043
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51901
diff changeset
578 class VolatileManager:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
579 """Manage temporary backups of volatile files during stream clone.
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
580
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
581 This class will keep open file handles for the volatile files, writing the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
582 smaller ones on disk if the number of open file handles grow too much.
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
583
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
584 This should be used as a Python context, the file handles and copies will
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
585 be discarded when exiting the context.
49222
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49037
diff changeset
586
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
587 The preservation can be done by calling the object on the real path
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
588 (encoded full path).
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
589
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
590 Valid filehandles for any file should be retrieved by calling `open(path)`.
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
591 """
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
592
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
593 # arbitrarily picked as "it seemed fine" and much higher than the current
52154
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52046
diff changeset
594 # usage. The Windows value of 2 is actually 1 file open at a time, due to
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52046
diff changeset
595 # the `flush_count = self.MAX_OPEN // 2` and `self.MAX_OPEN - 1` threshold
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52046
diff changeset
596 # for flushing to disk in __call__().
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52046
diff changeset
597 MAX_OPEN = 2 if pycompat.iswindows else 100
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
598
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
599 def __init__(self) -> None:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
600 self._counter = 0
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
601 self._volatile_fps = None
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
602 self._copies = None
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
603 self._dst_dir = None
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
604
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
605 def __enter__(self):
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
606 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
607 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
608 self._volatile_fps = {}
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
609 self._counter += 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
610 return self
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
611
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
612 def __exit__(self, *args, **kwars):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
613 """discard all backups"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
614 self._counter -= 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
615 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
616 for _size, fp in self._volatile_fps.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
617 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
618 self._volatile_fps = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
619 if self._copies is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
620 for tmp in self._copies.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
621 util.tryunlink(tmp)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
622 util.tryrmdir(self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
623 self._copies = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
624 self._dst_dir = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
625 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
626 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
627 assert self._dst_dir is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
628
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
629 def _init_tmp_copies(self) -> None:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
630 """prepare a temporary directory to save volatile files
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
631
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
632 This will be used as backup if we have too many files open"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
633 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
634 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
635 assert self._dst_dir is None
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
636 self._copies = {}
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
637 self._dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
638
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
639 def _flush_some_on_disk(self) -> None:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
640 """move some of the open files to tempory files on disk"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
641 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
642 self._init_tmp_copies()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
643 flush_count = self.MAX_OPEN // 2
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
644 for src, (size, fp) in sorted(self._volatile_fps.items())[:flush_count]:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
645 prefix = os.path.basename(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
646 fd, dst = pycompat.mkstemp(prefix=prefix, dir=self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
647 self._copies[src] = dst
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
648 os.close(fd)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
649 # we no longer hardlink, but on the other hand we rarely do this,
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
650 # and we do it for the smallest file only and not at all in the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
651 # common case.
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
652 with open(dst, 'wb') as bck:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
653 fp.seek(0)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
654 bck.write(fp.read())
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
655 del self._volatile_fps[src]
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
656 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
657
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
658 def _keep_one(self, src: bytes) -> int:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
659 """preserve an open file handle for a given path"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
660 # store the file quickly to ensure we close it if any error happens
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
661 _, fp = self._volatile_fps[src] = (None, open(src, 'rb'))
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
662 fp.seek(0, os.SEEK_END)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
663 size = fp.tell()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
664 self._volatile_fps[src] = (size, fp)
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
665 return size
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
666
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
667 def __call__(self, src: bytes) -> None:
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
668 """preserve the volatile file at src"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
669 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
670 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
671 self._flush_some_on_disk()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
672 self._keep_one(src)
50660
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50558
diff changeset
673
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
674 def try_keep(self, src: bytes) -> Optional[int]:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
675 """record a volatile file and returns it size
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
676
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
677 return None if the file does not exists.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
678
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
679 Used for cache file that are not lock protected.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
680 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
681 assert 0 < self._counter
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
682 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
683 self._flush_some_on_disk()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
684 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
685 return self._keep_one(src)
52665
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52391
diff changeset
686 except OSError as err:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
687 if err.errno not in (errno.ENOENT, errno.EPERM):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
688 raise
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
689 return None
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
690
52044
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52043
diff changeset
691 @contextlib.contextmanager
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52043
diff changeset
692 def open(self, src):
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
693 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
694 entry = self._volatile_fps.get(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
695 if entry is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
696 _size, fp = entry
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
697 fp.seek(0)
52044
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52043
diff changeset
698 yield fp
52046
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
699 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
700 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
701 actual_path = src
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
702 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
703 actual_path = self._copies.get(src, src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
704 with open(actual_path, 'rb') as fp:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52045
diff changeset
705 yield fp
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
706
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
707
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
708 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
709 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
710 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
711 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
712 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
713 }
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
714 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
715 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
716 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
717
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
718 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
719
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
720
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
721 def _emit2(repo, entries):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
722 """actually emit the stream bundle"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
723 vfsmap = _makemap(repo)
47050
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
724 # we keep repo.vfs out of the on purpose, ther are too many danger there
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
725 # (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
726 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
727 # this assert is duplicated (from _makemap) as author might think this is
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
728 # fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
729 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
730 raise error.ProgrammingError(
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
731 b'repo.vfs must not be added to vfsmap for security reasons'
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
732 )
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
733
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
734 # translate the vfs one
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
735 entries = [(vfs_key, vfsmap[vfs_key], e) for (vfs_key, e) in entries]
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
736 _test_sync_point_walk_1_2(repo)
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
737
50670
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
738 max_linkrev = len(repo)
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
739 file_count = totalfilesize = 0
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
740 with VolatileManager() as volatiles:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
741 # make sure we preserve volatile files
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
742 with util.nogc():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
743 # record the expected size of every file
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
744 for k, vfs, e in entries:
52389
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52388
diff changeset
745 e.preserve_volatiles(vfs, volatiles)
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
746 for f in e.files():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
747 file_count += 1
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
748 totalfilesize += f.file_size(vfs)
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
749
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
750 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
751 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
752 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
753 progress.update(0)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
754 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
755 # the first yield release the lock on the repository
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
756 yield file_count, totalfilesize
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
757 totalbytecount = 0
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
758
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
759 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
760 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
761 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
762 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
763 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
764 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
765 preserve_file_count=True,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
766 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
767 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
768 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
769 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
770 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
771 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
772 bytecount = 0
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
773 for chunk in stream:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
774 bytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
775 totalbytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
776 progress.update(totalbytecount)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
777 yield chunk
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
778 if bytecount != size:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
779 # Would most likely be caused by a race due to `hg
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
780 # strip` or a revlog split
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
781 msg = _(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
782 b'clone could only read %d bytes from %s, but '
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
783 b'expected %d bytes'
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
784 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
785 raise error.Abort(msg % (bytecount, name, size))
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
786
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
787
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
788 def _emit3(repo, entries) -> Iterator[bytes | None]:
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
789 """actually emit the stream bundle (v3)"""
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
790 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
791 # we keep repo.vfs out of the map on purpose, ther are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
792 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
793 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
794 # this assert is duplicated (from _makemap) as authors might think this is
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
795 # fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
796 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
797 raise error.ProgrammingError(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
798 b'repo.vfs must not be added to vfsmap for security reasons'
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
799 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
800
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
801 # translate the vfs once
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
802 # we only turn this into a list for the `_test_sync`, this is not ideal
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
803 base_entries = list(entries)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
804 _test_sync_point_walk_1_2(repo)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
805 entries = []
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
806 with VolatileManager() as volatiles:
52043
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51901
diff changeset
807 # make sure we preserve volatile files
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
808 for vfs_key, e in base_entries:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
809 vfs = vfsmap[vfs_key]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
810 any_files = True
51590
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
811 if e.maybe_volatile:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
812 any_files = False
52389
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52388
diff changeset
813 e.preserve_volatiles(vfs, volatiles)
51590
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
814 for f in e.files():
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
815 if f.is_volatile:
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
816 # record the expected size under lock
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51589
diff changeset
817 f.file_size(vfs)
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
818 any_files = True
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
819 if any_files:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
820 entries.append((vfs_key, vfsmap[vfs_key], e))
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
821
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
822 total_entry_count = len(entries)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
823
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
824 max_linkrev = len(repo)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
825 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
826 _(b'bundle'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
827 total=total_entry_count,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
828 unit=_(b'entry'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
829 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
830 progress.update(0)
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
831 # the first yield release the lock on the repository
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
832 yield None
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
833 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
834 yield util.uvarintencode(total_entry_count)
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
835
52388
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
836 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
837 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
838 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
839 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
840 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
841 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
842 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
843 yield util.uvarintencode(len(entry_streams))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
844 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
845 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
846 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
847 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
848 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
849 yield from stream
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52154
diff changeset
850 progress.increment()
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
851
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
852
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
853 def _test_sync_point_walk_1_2(repo):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
854 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
855
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
856 Triggered after gather entry, but before starting to process/preserve them
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
857 under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
858
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
859 (on v1 is triggered before the actual walk start)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
860 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
861
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
862
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
863 def _test_sync_point_walk_3(repo):
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
864 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
865
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
866 Triggered right before releasing the lock, but after computing what need
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
867 needed to compute under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
868 """
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
869
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
870
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
871 def _test_sync_point_walk_4(repo):
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
872 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
873
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
874 Triggered right after releasing the lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
875 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
876
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
877
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
878 # not really a StoreEntry, but close enough
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
879 class CacheEntry(store.SimpleStoreEntry):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
880 """Represent an entry for Cache files
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
881
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
882 It has special logic to preserve cache file early and accept optional
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
883 presence.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
884
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
885
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
886 (Yes... this is not really a StoreEntry, but close enough. We could have a
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
887 BaseEntry base class, bbut the store one would be identical)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
888 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
889
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
890 def __init__(self, entry_path) -> None:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
891 super().__init__(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
892 entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
893 # we will directly deal with that in `setup_cache_file`
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
894 is_volatile=True,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
895 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
896
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
897 def preserve_volatiles(self, vfs, volatiles) -> None:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
898 self._file_size = volatiles.try_keep(vfs.join(self._entry_path))
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
899 if self._file_size is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
900 self._files = []
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
901 else:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
902 assert self._is_volatile
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
903 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
904 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
905 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
906 file_size=self._file_size,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
907 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
908 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
909 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
910
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
911 def files(self) -> list[store.StoreFile]:
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
912 if self._files is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
913 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
914 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
915 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
916 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
917 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
918 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
919 return super().files()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
920
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
921
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
922 class CacheFile(store.StoreFile):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
923 # inform the "copy/hardlink" version that this file might be missing
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
924 # without consequences.
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
925 optional: bool = True
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
926
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
927
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
928 def _entries_walk(repo, includes, excludes, includeobsmarkers: bool):
50549
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
929 """emit a seris of files information useful to clone a repo
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
930
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
931 return (vfs-key, entry) iterator
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
932
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
933 Where `entry` is StoreEntry. (used even for cache entries)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
934 """
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
935 assert repo._currentlock(repo._lockref) is not None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
936
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
937 matcher = None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
938 if includes or excludes:
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
939 matcher = narrowspec.match(repo.root, includes, excludes)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
940
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
941 phase = not repo.publishing()
51588
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
942 # Python is getting crazy at all the small container we creates, disabling
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
943 # the gc while we do so helps performance a lot.
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
944 with util.nogc():
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
945 entries = _walkstreamfiles(
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
946 repo,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
947 matcher,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
948 phase=phase,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
949 obsolescence=includeobsmarkers,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
950 )
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
951 for entry in entries:
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
952 yield (_srcstore, entry)
50549
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
953
51588
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
954 for name in cacheutil.cachetocopy(repo):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
955 if repo.cachevfs.exists(name):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51586
diff changeset
956 # not really a StoreEntry, but close enough
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
957 yield (_srccache, CacheEntry(entry_path=name))
50550
43ed1f12b00a stream-clone: yield cache entry in `_entries_walk` too
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50549
diff changeset
958
50549
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50548
diff changeset
959
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
960 def generatev2(repo, includes, excludes, includeobsmarkers: bool):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
961 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
962
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
963 the data stream consists the following entries:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
964 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
965 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
966 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
967 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
968 5) N bytes containing the file data
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
969
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
970 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
971 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
972
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
973 with repo.lock():
47450
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
974 repo.ui.debug(b'scanning\n')
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
975
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
976 entries = _entries_walk(
47450
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
977 repo,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
978 includes=includes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
979 excludes=excludes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
980 includeobsmarkers=includeobsmarkers,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
981 )
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
982 chunks = _emit2(repo, entries)
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
983 first = next(chunks)
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
984 file_count, total_file_size = first
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
985 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
986 _test_sync_point_walk_4(repo)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
987
50665
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50664
diff changeset
988 return file_count, total_file_size, chunks
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
989
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
990
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
991 def generatev3(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
992 repo, includes, excludes, includeobsmarkers: bool
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
993 ) -> Iterator[bytes | None]:
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
994 """Emit content for version 3 of a streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
995
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
996 the data stream consists the following:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
997 1) A varint E containing the number of entries (can be 0), then E entries follow
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
998 2) For each entry:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
999 2.1) The number of files in this entry (can be 0, but typically 1 or 2)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1000 2.2) For each file:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1001 2.2.1) A char representing the file destination (eg: store or cache)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1002 2.2.2) A varint N containing the length of the filename
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1003 2.2.3) A varint M containing the length of file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1004 2.2.4) N bytes containing the filename (the internal, store-agnostic form)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1005 2.2.5) M bytes containing the file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1006
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1007 Returns the data iterator.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1008
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1009 XXX This format is experimental and subject to change. Here is a
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1010 XXX non-exhaustive list of things this format could do or change:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1011
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1012 - making it easier to write files in parallel
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1013 - holding the lock for a shorter time
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1014 - improving progress information
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1015 - ways to adjust the number of expected entries/files ?
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1016 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1017
51589
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51588
diff changeset
1018 # Python is getting crazy at all the small container we creates while
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51588
diff changeset
1019 # considering the files to preserve, disabling the gc while we do so helps
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51588
diff changeset
1020 # performance a lot.
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51588
diff changeset
1021 with repo.lock(), util.nogc():
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1022 repo.ui.debug(b'scanning\n')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1023
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1024 entries = _entries_walk(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1025 repo,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1026 includes=includes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1027 excludes=excludes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1028 includeobsmarkers=includeobsmarkers,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1029 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1030 chunks = _emit3(repo, list(entries))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1031 first = next(chunks)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1032 assert first is None
52390
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
1033 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52389
diff changeset
1034 _test_sync_point_walk_4(repo)
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1035
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1036 return chunks
50558
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50557
diff changeset
1037
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50557
diff changeset
1038
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1039 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1040 def nested(*ctxs):
39773
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1041 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1042 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1043 with this:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1044 if rest:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1045 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1046 yield
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39716
diff changeset
1047 else:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1048 yield
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1049
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1050
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1051 def consumev2(repo, fp, filecount: int, filesize: int) -> None:
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1052 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1053
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1054 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1055 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1056 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1057 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1058 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1059 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1060 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1061 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1062
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1063 start = util.timer()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1064 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1065 _(b'clone'), total=filesize, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1066 )
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
1067 progress.update(0)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1068
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1069 vfsmap = _makemap(repo)
47050
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1070 # we keep repo.vfs out of the on purpose, ther are too many danger
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1071 # there (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1072 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1073 # this assert is duplicated (from _makemap) as author might think this
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1074 # is fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1075 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1076 raise error.ProgrammingError(
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1077 b'repo.vfs must not be added to vfsmap for security reasons'
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1078 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1079
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1080 with repo.transaction(b'clone'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1081 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1082 with nested(*ctxs):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1083 for i in range(filecount):
35803
3ad3aaeb1134 streamclone: use readexactly when reading stream v2
Boris Feld <boris.feld@octobus.net>
parents: 35802
diff changeset
1084 src = util.readexactly(fp, 1)
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1085 vfs = vfsmap[src]
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1086 namelen = util.uvarintdecodestream(fp)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1087 datalen = util.uvarintdecodestream(fp)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1088
35803
3ad3aaeb1134 streamclone: use readexactly when reading stream v2
Boris Feld <boris.feld@octobus.net>
parents: 35802
diff changeset
1089 name = util.readexactly(fp, namelen)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1090
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1091 if repo.ui.debugflag:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1092 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1093 b'adding [%s] %s (%s)\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1094 % (src, name, util.bytecount(datalen))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1095 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1096
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1097 with vfs(name, b'w') as ofp:
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1098 for chunk in util.filechunkiter(fp, limit=datalen):
38355
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38197
diff changeset
1099 progress.increment(step=len(chunk))
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1100 ofp.write(chunk)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1101
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1102 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1103 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1104 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1105
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1106 elapsed = util.timer() - start
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1107 if elapsed <= 0:
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1108 elapsed = 0.001
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1109 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1110 _(b'transferred %s in %.1f seconds (%s/sec)\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1111 % (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1112 util.bytecount(progress.pos),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1113 elapsed,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1114 util.bytecount(progress.pos / elapsed),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1115 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1116 )
38379
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38366
diff changeset
1117 progress.complete()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1118
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1119
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1120 def consumev3(repo, fp) -> None:
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1121 """Apply the contents from a version 3 streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1122
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1123 Data is read from an object that only needs to provide a ``read(size)``
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1124 method.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1125 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1126 with repo.lock():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1127 start = util.timer()
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1128
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1129 entrycount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1130 repo.ui.status(_(b'%d entries to transfer\n') % (entrycount))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1131
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1132 progress = repo.ui.makeprogress(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1133 _(b'clone'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1134 total=entrycount,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1135 unit=_(b'entries'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1136 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1137 progress.update(0)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1138 bytes_transferred = 0
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1139
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1140 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1141 # we keep repo.vfs out of the on purpose, there are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1142 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1143 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1144 # this assert is duplicated (from _makemap) as authors might think this
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1145 # is fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1146 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1147 raise error.ProgrammingError(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1148 b'repo.vfs must not be added to vfsmap for security reasons'
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1149 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1150
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1151 with repo.transaction(b'clone'):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1152 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1153 with nested(*ctxs):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1154 for i in range(entrycount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1155 filecount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1156 if filecount == 0:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1157 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1158 repo.ui.debug(b'entry with no files [%d]\n' % (i))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1159 for i in range(filecount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1160 src = util.readexactly(fp, 1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1161 vfs = vfsmap[src]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1162 namelen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1163 datalen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1164
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1165 name = util.readexactly(fp, namelen)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1166
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1167 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1168 msg = b'adding [%s] %s (%s)\n'
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1169 msg %= (src, name, util.bytecount(datalen))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1170 repo.ui.debug(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1171 bytes_transferred += datalen
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1172
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1173 with vfs(name, b'w') as ofp:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1174 for chunk in util.filechunkiter(fp, limit=datalen):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1175 ofp.write(chunk)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1176 progress.increment(step=1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1177
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1178 # force @filecache properties to be reloaded from
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1179 # streamclone-ed file at next access
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1180 repo.invalidate(clearfilecache=True)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1181
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1182 elapsed = util.timer() - start
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1183 if elapsed <= 0:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1184 elapsed = 0.001
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1185 msg = _(b'transferred %s in %.1f seconds (%s/sec)\n')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1186 byte_count = util.bytecount(bytes_transferred)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1187 bytes_sec = util.bytecount(bytes_transferred / elapsed)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1188 msg %= (byte_count, elapsed, bytes_sec)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1189 repo.ui.status(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1190 progress.complete()
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1191
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1192
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1193 def applybundlev2(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1194 repo, fp, filecount: int, filesize: int, requirements: Iterable[bytes]
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1195 ) -> None:
39716
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1196 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1197
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1198 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1199 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1200 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
1201 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1202 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1203 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1204
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35495
diff changeset
1205 consumev2(repo, fp, filecount, filesize)
35804
2d3e486d09d0 streamclone: move requirement update into consumev2
Boris Feld <boris.feld@octobus.net>
parents: 35803
diff changeset
1206
48618
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
1207 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
1208 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47919
diff changeset
1209 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1210 )
39716
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38823
diff changeset
1211 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1212 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42823
diff changeset
1213 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
1214 scmutil.writereporequirements(repo)
48707
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48695
diff changeset
1215 nodemap.post_stream_cleanup(repo)
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1216
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1217
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1218 def applybundlev3(repo, fp, requirements: Iterable[bytes]) -> None:
50706
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1219 from . import localrepo
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1220
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1221 missingreqs = [r for r in requirements if r not in repo.supported]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1222 if missingreqs:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1223 msg = _(b'unable to apply stream clone: unsupported format: %s')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1224 msg %= b', '.join(sorted(missingreqs))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1225 raise error.Abort(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1226
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1227 consumev3(repo, fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1228
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1229 repo.requirements = new_stream_clone_requirements(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1230 repo.requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1231 requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1232 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1233 repo.svfs.options = localrepo.resolvestorevfsoptions(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1234 repo.ui, repo.requirements, repo.features
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1235 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1236 scmutil.writereporequirements(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1237 nodemap.post_stream_cleanup(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1238
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50675
diff changeset
1239
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1240 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress) -> bool:
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1241 hardlink = [True]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1242
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1243 def copy_used():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1244 hardlink[0] = False
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1245 progress.topic = _(b'copying')
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1246
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1247 for k, path, optional in entries:
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1248 src_vfs = src_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1249 dst_vfs = dst_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1250 src_path = src_vfs.join(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1251 dst_path = dst_vfs.join(path)
47863
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1252 # We cannot use dirname and makedirs of dst_vfs here because the store
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1253 # encoding confuses them. See issue 6581 for details.
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1254 dirname = os.path.dirname(dst_path)
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1255 if not os.path.exists(dirname):
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47787
diff changeset
1256 util.makedirs(dirname)
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1257 dst_vfs.register_file(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1258 # XXX we could use the #nb_bytes argument.
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1259 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1260 util.copyfile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1261 src_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1262 dst_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1263 hardlink=hardlink[0],
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1264 no_hardlink_cb=copy_used,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1265 check_fs_hardlink=False,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1266 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1267 except FileNotFoundError:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1268 if not optional:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1269 raise
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1270 progress.increment()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1271 return hardlink[0]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1272
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1273
52713
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52709
diff changeset
1274 def local_copy(src_repo, dest_repo) -> None:
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1275 """copy all content from one local repository to another
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1276
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1277 This is useful for local clone"""
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1278 src_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1279 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1280 for r in src_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1281 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1282 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1283 dest_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1284 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1285 for r in dest_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1286 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1287 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1288 assert src_store_requirements == dest_store_requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1289
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1290 with dest_repo.lock():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1291 with src_repo.lock():
47454
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1292 # bookmark is not integrated to the streaming as it might use the
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1293 # `repo.vfs` and they are too many sentitive data accessible
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1294 # through `repo.vfs` to expose it to streaming clone.
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1295 src_book_vfs = bookmarks.bookmarksvfs(src_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1296 srcbookmarks = src_book_vfs.join(b'bookmarks')
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1297 bm_count = 0
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1298 if os.path.exists(srcbookmarks):
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1299 bm_count = 1
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1300
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1301 entries = _entries_walk(
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1302 src_repo,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1303 includes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1304 excludes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1305 includeobsmarkers=True,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1306 )
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1307 entries = list(entries)
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1308 src_vfs_map = _makemap(src_repo)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1309 dest_vfs_map = _makemap(dest_repo)
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1310 total_files = sum(len(e[1].files()) for e in entries) + bm_count
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1311 progress = src_repo.ui.makeprogress(
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1312 topic=_(b'linking'),
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1313 total=total_files,
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1314 unit=_(b'files'),
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1315 )
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1316 # copy files
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1317 #
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1318 # We could copy the full file while the source repository is locked
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1319 # and the other one without the lock. However, in the linking case,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1320 # this would also requires checks that nobody is appending any data
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1321 # to the files while we do the clone, so this is not done yet. We
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1322 # could do this blindly when copying files.
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1323 files = [
52391
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52390
diff changeset
1324 (vfs_key, f.unencoded_path, f.optional)
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1325 for vfs_key, e in entries
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1326 for f in e.files()
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1327 ]
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1328 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1329
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1330 # copy bookmarks over
47454
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1331 if bm_count:
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1332 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47453
diff changeset
1333 dstbookmarks = dst_book_vfs.join(b'bookmarks')
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1334 util.copyfile(srcbookmarks, dstbookmarks)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1335 progress.complete()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1336 if hardlink:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1337 msg = b'linked %d files\n'
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1338 else:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1339 msg = b'copied %d files\n'
50663
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50660
diff changeset
1340 src_repo.ui.debug(msg % total_files)
47453
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1341
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1342 with dest_repo.transaction(b"localclone") as tr:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1343 dest_repo.store.write(tr)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1344
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47450
diff changeset
1345 # clean up transaction file as they do not make sense
50288
d89eecf9605e undo-files: no longer pass the `repo` to `cleanup_undo_files`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50286
diff changeset
1346 transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)