Mercurial > public > mercurial-scm > hg

annotate mercurial/streamclone.py @ 52971:c636f2835c95

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
stream-bundle: make sure to open the file in binary mode Windows need an additional flag to open the file properly.
author Pierre-Yves David <pierre-yves.david@octobus.net>
date Wed, 19 Feb 2025 01:13:00 +0100
parents e4ff37b5317c
children 5480647c2964
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
51859
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
8 from __future__ import annotations
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
52922
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
10 import collections
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
11 import contextlib
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
12 import errno
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
13 import os
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
14 import struct
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
15 import threading
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
16
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
17 from typing import (
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
18 Callable,
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
19 Iterable,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
20 Iterator,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
21 Optional,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
22 Set,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
23 Tuple,
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
24 Type,
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
25 )
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
26
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
27 from .i18n import _
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
28 from .interfaces import repository
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
29 from . import (
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
30 bookmarks,
50520
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
31 bundle2 as bundle2mod,
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
32 cacheutil,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
33 error,
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
34 narrowspec,
32744
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
35 phases,
38164
aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path
Yuya Nishihara <yuya@tcha.org>
parents: 37638
diff changeset
36 pycompat,
46626
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
37 requirements as requirementsmod,
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
38 scmutil,
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
39 store,
50286
3d0b5760851c undo-files: move the undo cleanup code in the transaction module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50279
diff changeset
40 transaction,
26442
ef8d27f53204 streamclone: move stream_in() from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26441
diff changeset
41 util,
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
42 )
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
43 from .revlogutils import (
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
44 nodemap,
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
45 )
26441
56527b886d1d streamclone: move applystreamclone() from localrepo.py
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
46
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
47
52913
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
48 # Number arbitrarily picked, feel free to change them (but the LOW one)
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
49 #
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
50 # update the configuration documentation if you touch this.
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
51 DEFAULT_NUM_WRITER = {
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
52 scmutil.RESOURCE_LOW: 1,
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
53 scmutil.RESOURCE_MEDIUM: 2,
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
54 scmutil.RESOURCE_HIGH: 4,
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
55 }
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
56
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
57
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
58 # Number arbitrarily picked, feel free to adjust them. Do update the
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
59 # documentation if you do so
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
60 DEFAULT_MEMORY_TARGET = {
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
61 scmutil.RESOURCE_LOW: 100 * (2**20), # 100 MB
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
62 scmutil.RESOURCE_MEDIUM: 2**30, # 1 GB
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
63 scmutil.RESOURCE_HIGH: None,
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
64 }
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
65
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
66
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
67 def new_stream_clone_requirements(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
68 default_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
69 streamed_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
70 ) -> Set[bytes]:
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
71 """determine the final set of requirement for a new stream clone
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
72
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
73 this method combine the "default" requirements that a new repository would
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
74 use with the constaint we get from the stream clone content. We keep local
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
75 configuration choice when possible.
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
76 """
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
77 requirements = set(default_requirements)
48601
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
78 requirements -= requirementsmod.STREAM_FIXED_REQUIREMENTS
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
79 requirements.update(streamed_requirements)
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
80 return requirements
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
81
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
82
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
83 def streamed_requirements(repo) -> Set[bytes]:
48597
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
84 """the set of requirement the new clone will have to support
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
85
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
86 This is used for advertising the stream options and to generate the actual
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
87 stream content."""
48601
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
88 requiredformats = (
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
89 repo.requirements & requirementsmod.STREAM_FIXED_REQUIREMENTS
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48600
diff changeset
90 )
48597
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
91 return requiredformats
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
92
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
93
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
94 def canperformstreamclone(pullop, bundle2: bool = False):
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
95 """Whether it is possible to perform a streaming clone as part of pull.
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
96
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
97 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
98 bundle2 and only through bundle2.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
99
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
100 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
101 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
102 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
103 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
104 """
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
105 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
106 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
107
50519
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
108 # should we consider streaming clone at all ?
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
109 streamrequested = pullop.streamclonerequested
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
110 # If we don't have a preference, let the server decide for us. This
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
111 # likely only comes into play in LANs.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
112 if streamrequested is None:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
113 # The server can advertise whether to prefer streaming clone.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
114 streamrequested = remote.capable(b'stream-preferred')
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
115 if not streamrequested:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
116 return False, None
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50518
diff changeset
117
50517
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
118 # Streaming clone only works on an empty destination repository
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
119 if len(repo):
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
120 return False, None
0558866957fa stream-clone: bail-out earlier if destination repo is not empty
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50513
diff changeset
121
50518
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
122 # Streaming clone only works if all data is being requested.
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
123 if pullop.heads:
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
124 return False, None
f697af015683 stream-clone: bail-out earlier if pull is partial
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50517
diff changeset
125
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
126 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
127 if pullop.canusebundle2:
50520
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
128 local_caps = bundle2mod.getrepocaps(repo, role=b'client')
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
129 local_supported = set(local_caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
130 remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50519
diff changeset
131 bundle2supported = bool(local_supported & remote_supported)
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
132 # else
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
133 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
134 # the versions we support. Fall back and possibly allow legacy.
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
135
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
136 # Ensures legacy code path uses available bundle2.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
137 if bundle2supported and not bundle2:
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
138 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
139 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
140 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone
Boris Feld <boris.feld@octobus.net>
parents: 35756
diff changeset
141 return False, None
26467
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26466
diff changeset
142
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
143 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
144 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
145 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
146 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
147 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
148 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
149 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
150 requirements = set()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
151 if remote.capable(b'stream'):
46626
ee91966aec0f requirements: add constant for revlog v1 requirement
Rapha?l Gom?s <rgomes@octobus.net>
parents: 45106
diff changeset
152 requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
153 else:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
154 streamreqs = remote.capable(b'streamreqs')
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
155 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
156 if not streamreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
157 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
158 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
159 b'warning: stream clone requested but server has them '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
160 b'disabled\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
161 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
162 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
163 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
164
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
165 streamreqs = set(streamreqs.split(b','))
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
166 # Server requires something we don't support. Bail.
48670
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
167 missingreqs = streamreqs - repo.supported
32259
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30975
diff changeset
168 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed
Siddharth Agarwal <sid0@fb.com>
parents: 30975
diff changeset
169 pullop.repo.ui.warn(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
170 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
171 b'warning: stream clone requested but client is missing '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
172 b'requirements: %s\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
173 )
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
174 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
175 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
176 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
177 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
178 b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
179 b'for more information)\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
180 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
181 )
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
182 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
183 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
184
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
185 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
186
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
187
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
188 def maybeperformlegacystreamclone(pullop) -> None:
26462
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
189 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
190
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
191 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
192 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
193
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
194 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
195 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26461
diff changeset
196 """
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
197 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
198
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
199 supported, requirements = canperformstreamclone(pullop)
26458
362793295640 streamclone: refactor maybeperformstreamclone to take a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26447
diff changeset
200
26446
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
201 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
202 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26445
diff changeset
203
26466
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
204 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
205 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26462
diff changeset
206
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
207 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
208 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
209 rbranchmap = None
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
210 if remote.capable(b'branchmap'):
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
211 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
212 rbranchmap = e.callcommand(b'branchmap', {}).result()
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
213
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
214 repo.ui.status(_(b'streaming all changes\n'))
26470
4b5647d9ee13 streamclone: move "streaming all changes" message location
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26469
diff changeset
215
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
216 with remote.commandexecutor() as e:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
217 fp = e.callcommand(b'stream_out', {}).result()
37638
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
218
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
219 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
220 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
221 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35804
diff changeset
222 # doesn't matter.
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
223 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
224 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
225 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
226 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
227 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
228 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
229 )
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
230 if resp == 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
231 raise error.Abort(_(b'operation forbidden by server'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
232 elif resp == 2:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
233 raise error.Abort(_(b'locking the remote repository failed'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
234 elif resp != 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
235 raise error.Abort(_(b'the server sent an unknown error code'))
26459
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26458
diff changeset
236
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
237 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
238 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
239 filecount, bytecount = map(int, l.split(b' ', 1))
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
240 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
241 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
242 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
243 )
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
244
27850
49cfddbf54ba with: use context manager in maybeperformlegacystreamclone
Bryan O'Sullivan <bryano@fb.com>
parents: 27845
diff changeset
245 with repo.lock():
26468
19bbd53af46d streamclone: move payload header line consumption
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26467
diff changeset
246 consumev1(repo, fp, filecount, bytecount)
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
247 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
248 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
249 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
250 )
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
251 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
252 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
253 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
254 scmutil.writereporequirements(repo)
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
255 nodemap.post_stream_cleanup(repo)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
256
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
257 if rbranchmap:
41615
328ca3b9e545 branchmap: encapsulate cache updating in the map itself
Martijn Pieters <mj@octobus.net>
parents: 40494
diff changeset
258 repo._branchcaches.replace(repo, rbranchmap)
26461
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
259
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26460
diff changeset
260 repo.invalidate()
26445
f134fb33c906 streamclone: move streaming clone logic from localrepo
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26444
diff changeset
261
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
262
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
263 def allowservergeneration(repo) -> bool:
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
264 """Whether streaming clones are allowed from the server."""
40028
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
265 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
266 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39850
diff changeset
267
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
268 if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
32744
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
269 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
270
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
271 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
272 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
273 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
274 if secret:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
275 return repo.ui.configbool(b'server', b'uncompressedallowsecret')
32744
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
276
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32259
diff changeset
277 return True
26444
623743010133 streamclone: move _allowstream() from wireproto
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26443
diff changeset
278
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
279
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
280 # This is it's own function so extensions can override it.
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
281 def _walkstreamfiles(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
282 repo, matcher=None, phase: bool = False, obsolescence: bool = False
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
283 ):
50511
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50509
diff changeset
284 return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
285
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
286
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
287 def _report_transferred(
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
288 repo, start_time: float, file_count: int, byte_count: int
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
289 ):
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
290 """common utility to report time it took to apply the stream bundle"""
52700
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
291 elapsed = util.timer() - start_time
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
292 if elapsed <= 0:
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
293 elapsed = 0.001
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
294 m = _(b'stream-cloned %d files / %s in %.1f seconds (%s/sec)\n')
52700
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
295 m %= (
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
296 file_count,
52700
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
297 util.bytecount(byte_count),
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
298 elapsed,
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
299 util.bytecount(byte_count / elapsed),
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
300 )
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
301 repo.ui.status(m)
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
302
061bfd699a56 stream: use an utility function to report transferred bytes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52696
diff changeset
303
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
304 def generatev1(repo) -> tuple[int, int, Iterator[bytes]]:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
305 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
306
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
307 This returns a 3-tuple of (file count, byte size, data iterator).
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
308
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
309 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
310 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
311 delimited by a null byte.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
312
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
313 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
314 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
315
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
316 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
317 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
318 for adding it.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
319
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
320 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
321 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
322 """
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
323 entries = []
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
324 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
325 # Get consistent snapshot of repo, lock during scan.
27845
7417e1c10253 with: use context manager in streamclone generatev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27794
diff changeset
326 with repo.lock():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
327 repo.ui.debug(b'scanning\n')
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
328 _test_sync_point_walk_1_2(repo)
50468
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50288
diff changeset
329 for entry in _walkstreamfiles(repo):
50469
9fdc28e21b68 store: introduce a EntryFile object to actually access file info
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50468
diff changeset
330 for f in entry.files():
50474
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
331 file_size = f.file_size(repo.store.vfs)
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
332 if file_size:
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
333 entries.append((f.unencoded_path, file_size))
4cbdfab6f812 store: lazily get file size on demand for the fncache case
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
334 total_bytes += file_size
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
335 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
336 _test_sync_point_walk_4(repo)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
337
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
338 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
339 b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
340 )
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
341
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
342 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
343 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
344
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
345 def emitrevlogdata() -> Iterator[bytes]:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
346 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
347 if debugflag:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
348 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
349 # partially encode name over the wire for backwards compat
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
350 yield b'%s\0%d\n' % (store.encodedir(name), size)
33411
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
351 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33410
diff changeset
352 # trusted by the local repo) and expensive
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
353 with svfs(name, b'rb', auditpath=False) as fp:
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
354 if size <= 65536:
33258
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False'
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 33257
diff changeset
355 yield fp.read(size)
33410
c784308305c6 streamclone: close large revlog files explicitly in generatev1()
Yuya Nishihara <yuya@tcha.org>
parents: 33258
diff changeset
356 else:
52644
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52640
diff changeset
357 yield from util.filechunkiter(fp, limit=size)
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
358
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
359 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
360
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
361
52691
279e217d6041 typing: lock in the new type annotations detected with the pyupgrade changes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52644
diff changeset
362 def generatev1wireproto(repo) -> Iterator[bytes]:
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
363 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
364
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
365 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
366 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
367 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
368
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
369 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
370 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
371 a permissions error for the server process).
26469
fb743268510e streamclone: move payload header generation into own function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26468
diff changeset
372 """
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
373 if not allowservergeneration(repo):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
374 yield b'1\n'
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
375 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
376
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
377 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
378 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
379 except error.LockError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
380 yield b'2\n'
35491
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
381 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
382
ded3a63f305b streamclone: move wire protocol status code from wireproto command
Gregory Szorc <gregory.szorc@gmail.com>
parents: 33499
diff changeset
383 # Indicates successful response.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
384 yield b'0\n'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
385 yield b'%d %d\n' % (filecount, bytecount)
52644
e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52640
diff changeset
386 yield from it
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
387
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
388
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
389 def generatebundlev1(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
390 repo, compression: bytes = b'UN'
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
391 ) -> tuple[Set[bytes], Iterator[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
392 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
393
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
394 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
395 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
396
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
397 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
398 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
399
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
400 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
401 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
402
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
403 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
404 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
405 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
406 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
407
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
408 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
409 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
410
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
411 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
412 """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
413 if compression != b'UN':
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
414 raise ValueError('we do not support the compression argument yet')
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
415
48597
8475a1364909 stream-clone: factor computation of requirement of a stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48596
diff changeset
416 requirements = streamed_requirements(repo)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
417 requires = b','.join(sorted(requirements))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
418
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
419 def gen() -> Iterator[bytes]:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
420 yield b'HGS1'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
421 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
422
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
423 filecount, bytecount, it = generatev1(repo)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
424 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
425 _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
426 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
427
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
428 yield struct.pack(b'>QQ', filecount, bytecount)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
429 yield struct.pack(b'>H', len(requires) + 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
430 yield requires + b'\0'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
431
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
432 # This is where we'll add compression in the future.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
433 assert compression == b'UN'
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
434
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
435 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
436 _(b'bundle'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
437 )
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
438 progress.update(0)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
439
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
440 for chunk in it:
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
441 progress.increment(step=len(chunk))
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
442 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
443
38373
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38360
diff changeset
444 progress.complete()
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
445
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
446 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
447
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
448
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
449 def consumev1(repo, fp, filecount: int, bytecount: int) -> None:
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
450 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
451
30332
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29919
diff changeset
452 This takes the output from "stream_out" and applies it to the specified
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
453 repository.
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
454
30332
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29919
diff changeset
455 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 29919
diff changeset
456 handled by this function.
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
457 """
27859
f55a5ace8e69 with: use context manager in streamclone consumev1
Bryan O'Sullivan <bryano@fb.com>
parents: 27850
diff changeset
458 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
459 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
460 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
461 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
462 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
463 progress = repo.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
464 _(b'clone'), total=bytecount, unit=_(b'bytes')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
465 )
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
466 progress.update(0)
30975
22fbca1d11ed mercurial: switch to util.timer for all interval timings
Simon Farnsworth <simonfar@fb.com>
parents: 30332
diff changeset
467 start = util.timer()
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
468
29919
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
469 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
470 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
471 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
472 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
473 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
474 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
475 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
476 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
477 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
478 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
479 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
480 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
481 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
482
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
483 total_file_count = 0
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
484 with repo.transaction(b'clone'):
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
485 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
49284
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 49190
diff changeset
486 for i in range(filecount):
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
487 # XXX doesn't support '\n' or '\r' in filenames
50925
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50703
diff changeset
488 if hasattr(fp, 'readline'):
50663
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
489 l = fp.readline()
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
490 else:
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
491 # inline clonebundles use a chunkbuffer, so no readline
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
492 # --> this should be small anyway, the first line
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
493 # only contains the size of the bundle
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
494 l_buf = []
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
495 while not (l_buf and l_buf[-1] == b'\n'):
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
496 l_buf.append(fp.read(1))
60f9602b413e clonebundles: add support for inline (streaming) clonebundles
Mathias De Mare <mathias.de_mare@nokia.com>
parents: 50637
diff changeset
497 l = b''.join(l_buf)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
498 try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
499 name, size = l.split(b'\0', 1)
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
500 size = int(size)
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
501 except (ValueError, TypeError):
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
502 raise error.ResponseError(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
503 _(b'unexpected response from remote server:'), l
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
504 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
505 if repo.ui.debugflag:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
506 repo.ui.debug(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
507 b'adding %s (%s)\n' % (name, util.bytecount(size))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
508 )
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
509 # for backwards compat, name was partially encoded
27897
2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27896
diff changeset
510 path = store.decodedir(name)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
511 with repo.svfs(path, b'w', backgroundclose=True) as ofp:
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
512 total_file_count += 1
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
513 for chunk in util.filechunkiter(fp, limit=size):
38349
e59eaf51cc0d streamclone: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38164
diff changeset
514 progress.increment(step=len(chunk))
27896
1d29893240cc streamclone: indent code
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27882
diff changeset
515 ofp.write(chunk)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
516
29919
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
517 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
518 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29917
diff changeset
519 repo.invalidate(clearfilecache=True)
26443
d947086d8973 streamclone: move code out of exchange.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26442
diff changeset
520
38373
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38360
diff changeset
521 progress.complete()
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
522 _report_transferred(repo, start, total_file_count, bytecount)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
523
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
524
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
525 def readbundle1header(fp) -> tuple[int, int, Set[bytes]]:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
526 compression = fp.read(2)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
527 if compression != b'UN':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
528 raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
529 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
530 b'only uncompressed stream clone bundles are '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
531 b'supported; got %s'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
532 )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
533 % compression
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
534 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
535
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
536 filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
537 requireslen = struct.unpack(b'>H', fp.read(2))[0]
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
538 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
539
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
540 if not requires.endswith(b'\0'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
541 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
542 _(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
543 b'malformed stream clone bundle: '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
544 b'requirements not properly encoded'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
545 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
546 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
547
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
548 requirements = set(requires.rstrip(b'\0').split(b','))
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
549
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
550 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
551
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
552
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
553 def applybundlev1(repo, fp) -> None:
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
554 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
555
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
556 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
557 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
558 """
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
559 if len(repo):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
560 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
561 _(b'cannot apply stream clone bundle on non-empty repo')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
562 )
27882
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
563
319b0bf6ecc9 streamclone: extract code for reading header fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27870
diff changeset
564 filecount, bytecount, requirements = readbundle1header(fp)
48670
6d2ddea0721a stream-clone: filter possible missing requirements using all supported one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
565 missingreqs = requirements - repo.supported
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
566 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
567 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
568 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
569 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
570 )
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
571
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
572 consumev1(repo, fp, filecount, bytecount)
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
573 nodemap.post_stream_cleanup(repo)
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
574
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
575
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
576 class streamcloneapplier:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
577 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
578
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
579 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
580 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
581 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
582
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
583 def __init__(self, fh) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
584 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
585
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
586 def apply(self, repo) -> None:
26755
bb0b955d050d streamclone: support for producing and consuming stream clone bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26587
diff changeset
587 return applybundlev1(repo, self._fh)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
588
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
589
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
590 # type of file to stream
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
591 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
592 _filefull = 1 # full snapshot file
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
593
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
594 # Source of the file
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
595 _srcstore = b's' # store (svfs)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
596 _srccache = b'c' # cache (cache)
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
597
51699
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51696
diff changeset
598
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
599 # This is it's own function so extensions can override it.
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
600 def _walkstreamfullstorefiles(repo) -> list[bytes]:
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
601 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
602 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
603 if not repo.publishing():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
604 fnames.append(b'phaseroots')
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
605 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
606
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
607
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
608 def _filterfull(entry, copy, vfsmap):
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
609 """actually copy the snapshot files"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
610 src, name, ftype, data = entry
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
611 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
612 return entry
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
613 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
614
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
615
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51859
diff changeset
616 class VolatileManager:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
617 """Manage temporary backups of volatile files during stream clone.
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
618
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
619 This class will keep open file handles for the volatile files, writing the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
620 smaller ones on disk if the number of open file handles grow too much.
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
621
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
622 This should be used as a Python context, the file handles and copies will
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
623 be discarded when exiting the context.
49190
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48946
diff changeset
624
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
625 The preservation can be done by calling the object on the real path
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
626 (encoded full path).
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
627
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
628 Valid filehandles for any file should be retrieved by calling `open(path)`.
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
629 """
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
630
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
631 # arbitrarily picked as "it seemed fine" and much higher than the current
52184
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
632 # usage. The Windows value of 2 is actually 1 file open at a time, due to
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
633 # the `flush_count = self.MAX_OPEN // 2` and `self.MAX_OPEN - 1` threshold
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
634 # for flushing to disk in __call__().
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 52015
diff changeset
635 MAX_OPEN = 2 if pycompat.iswindows else 100
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
636
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
637 def __init__(self) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
638 self._counter = 0
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
639 self._volatile_fps = None
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
640 self._copies = None
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
641 self._dst_dir = None
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
642
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
643 def __enter__(self):
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
644 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
645 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
646 self._volatile_fps = {}
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
647 self._counter += 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
648 return self
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
649
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
650 def __exit__(self, *args, **kwars):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
651 """discard all backups"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
652 self._counter -= 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
653 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
654 for _size, fp in self._volatile_fps.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
655 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
656 self._volatile_fps = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
657 if self._copies is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
658 for tmp in self._copies.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
659 util.tryunlink(tmp)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
660 util.tryrmdir(self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
661 self._copies = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
662 self._dst_dir = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
663 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
664 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
665 assert self._dst_dir is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
666
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
667 def _init_tmp_copies(self) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
668 """prepare a temporary directory to save volatile files
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
669
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
670 This will be used as backup if we have too many files open"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
671 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
672 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
673 assert self._dst_dir is None
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
674 self._copies = {}
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
675 self._dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
676
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
677 def _flush_some_on_disk(self) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
678 """move some of the open files to tempory files on disk"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
679 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
680 self._init_tmp_copies()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
681 flush_count = self.MAX_OPEN // 2
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
682 for src, (size, fp) in sorted(self._volatile_fps.items())[:flush_count]:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
683 prefix = os.path.basename(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
684 fd, dst = pycompat.mkstemp(prefix=prefix, dir=self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
685 self._copies[src] = dst
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
686 os.close(fd)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
687 # we no longer hardlink, but on the other hand we rarely do this,
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
688 # and we do it for the smallest file only and not at all in the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
689 # common case.
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
690 with open(dst, 'wb') as bck:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
691 fp.seek(0)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
692 bck.write(fp.read())
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
693 del self._volatile_fps[src]
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
694 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
695
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
696 def _keep_one(self, src: bytes) -> int:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
697 """preserve an open file handle for a given path"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
698 # store the file quickly to ensure we close it if any error happens
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
699 _, fp = self._volatile_fps[src] = (None, open(src, 'rb'))
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
700 fp.seek(0, os.SEEK_END)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
701 size = fp.tell()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
702 self._volatile_fps[src] = (size, fp)
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
703 return size
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
704
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
705 def __call__(self, src: bytes) -> None:
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
706 """preserve the volatile file at src"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
707 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
708 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
709 self._flush_some_on_disk()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
710 self._keep_one(src)
50627
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50521
diff changeset
711
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
712 def try_keep(self, src: bytes) -> Optional[int]:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
713 """record a volatile file and returns it size
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
714
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
715 return None if the file does not exists.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
716
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
717 Used for cache file that are not lock protected.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
718 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
719 assert 0 < self._counter
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
720 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
721 self._flush_some_on_disk()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
722 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
723 return self._keep_one(src)
52640
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52359
diff changeset
724 except OSError as err:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
725 if err.errno not in (errno.ENOENT, errno.EPERM):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
726 raise
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
727 return None
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
728
52013
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
729 @contextlib.contextmanager
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
730 def open(self, src):
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
731 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
732 entry = self._volatile_fps.get(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
733 if entry is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
734 _size, fp = entry
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
735 fp.seek(0)
52013
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
736 yield fp
52015
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
737 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
738 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
739 actual_path = src
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
740 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
741 actual_path = self._copies.get(src, src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
742 with open(actual_path, 'rb') as fp:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52014
diff changeset
743 yield fp
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
744
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
745
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
746 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
747 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
748 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
749 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
750 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
751 }
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
752 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
753 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
754 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
755
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
756 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
757
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
758
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
759 def _emit2(repo, entries):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
760 """actually emit the stream bundle"""
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
761 vfsmap = _makemap(repo)
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
762 # we keep repo.vfs out of the on purpose, ther are too many danger there
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
763 # (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
764 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
765 # this assert is duplicated (from _makemap) as author might think this is
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
766 # fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
767 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
768 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
769 'repo.vfs must not be added to vfsmap for security reasons'
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
770 )
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
771
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
772 # translate the vfs one
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
773 entries = [(vfs_key, vfsmap[vfs_key], e) for (vfs_key, e) in entries]
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
774 _test_sync_point_walk_1_2(repo)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
775
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
776 max_linkrev = len(repo)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
777 file_count = totalfilesize = 0
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
778 with VolatileManager() as volatiles:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
779 # make sure we preserve volatile files
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
780 with util.nogc():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
781 # record the expected size of every file
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
782 for k, vfs, e in entries:
52357
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52356
diff changeset
783 e.preserve_volatiles(vfs, volatiles)
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
784 for f in e.files():
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
785 file_count += 1
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
786 totalfilesize += f.file_size(vfs)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
787
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
788 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
789 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
790 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
791 progress.update(0)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
792 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
793 # the first yield release the lock on the repository
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
794 yield file_count, totalfilesize
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
795 totalbytecount = 0
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
796
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
797 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
798 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
799 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
800 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
801 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
802 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
803 preserve_file_count=True,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
804 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
805 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
806 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
807 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
808 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
809 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
810 bytecount = 0
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
811 for chunk in stream:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
812 bytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
813 totalbytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
814 progress.update(totalbytecount)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
815 yield chunk
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
816 if bytecount != size:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
817 # Would most likely be caused by a race due to `hg
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
818 # strip` or a revlog split
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
819 msg = _(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
820 b'clone could only read %d bytes from %s, but '
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
821 b'expected %d bytes'
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
822 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
823 raise error.Abort(msg % (bytecount, name, size))
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
824
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
825
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
826 def _emit3(repo, entries) -> Iterator[bytes | None]:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
827 """actually emit the stream bundle (v3)"""
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
828 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
829 # we keep repo.vfs out of the map on purpose, ther are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
830 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
831 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
832 # this assert is duplicated (from _makemap) as authors might think this is
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
833 # fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
834 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
835 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
836 'repo.vfs must not be added to vfsmap for security reasons'
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
837 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
838
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
839 # translate the vfs once
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
840 # we only turn this into a list for the `_test_sync`, this is not ideal
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
841 base_entries = list(entries)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
842 _test_sync_point_walk_1_2(repo)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
843 entries = []
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
844 with VolatileManager() as volatiles:
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51859
diff changeset
845 # make sure we preserve volatile files
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
846 for vfs_key, e in base_entries:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
847 vfs = vfsmap[vfs_key]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
848 any_files = True
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
849 if e.maybe_volatile:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
850 any_files = False
52357
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52356
diff changeset
851 e.preserve_volatiles(vfs, volatiles)
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
852 for f in e.files():
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
853 if f.is_volatile:
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
854 # record the expected size under lock
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51548
diff changeset
855 f.file_size(vfs)
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
856 any_files = True
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
857 if any_files:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
858 entries.append((vfs_key, vfsmap[vfs_key], e))
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
859
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
860 total_entry_count = len(entries)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
861
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
862 max_linkrev = len(repo)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
863 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
864 _(b'bundle'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
865 total=total_entry_count,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
866 unit=_(b'entry'),
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
867 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
868 progress.update(0)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
869 # the first yield release the lock on the repository
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
870 yield None
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
871 with progress:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
872 yield util.uvarintencode(total_entry_count)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
873
52356
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
874 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
875 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
876 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
877 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
878 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
879 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
880 )
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
881 yield util.uvarintencode(len(entry_streams))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
882 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
883 yield src
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
884 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
885 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
886 yield name
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
887 yield from stream
46574e588017 stream: start the volatile manager sooner during emission
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52184
diff changeset
888 progress.increment()
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
889
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
890
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
891 def _test_sync_point_walk_1_2(repo):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
892 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
893
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
894 Triggered after gather entry, but before starting to process/preserve them
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
895 under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
896
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
897 (on v1 is triggered before the actual walk start)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
898 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
899
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
900
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
901 def _test_sync_point_walk_3(repo):
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
902 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
903
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
904 Triggered right before releasing the lock, but after computing what need
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
905 needed to compute under lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
906 """
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
907
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
908
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
909 def _test_sync_point_walk_4(repo):
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
910 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
911
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
912 Triggered right after releasing the lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
913 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
914
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
915
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
916 # not really a StoreEntry, but close enough
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
917 class CacheEntry(store.SimpleStoreEntry):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
918 """Represent an entry for Cache files
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
919
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
920 It has special logic to preserve cache file early and accept optional
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
921 presence.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
922
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
923
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
924 (Yes... this is not really a StoreEntry, but close enough. We could have a
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
925 BaseEntry base class, bbut the store one would be identical)
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
926 """
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
927
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
928 def __init__(self, entry_path) -> None:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
929 super().__init__(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
930 entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
931 # we will directly deal with that in `setup_cache_file`
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
932 is_volatile=True,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
933 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
934
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
935 def preserve_volatiles(self, vfs, volatiles) -> None:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
936 self._file_size = volatiles.try_keep(vfs.join(self._entry_path))
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
937 if self._file_size is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
938 self._files = []
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
939 else:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
940 assert self._is_volatile
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
941 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
942 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
943 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
944 file_size=self._file_size,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
945 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
946 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
947 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
948
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
949 def files(self) -> list[store.StoreFile]:
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
950 if self._files is None:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
951 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
952 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
953 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
954 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
955 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
956 ]
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
957 return super().files()
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
958
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
959
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
960 class CacheFile(store.StoreFile):
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
961 # inform the "copy/hardlink" version that this file might be missing
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
962 # without consequences.
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
963 optional: bool = True
46986
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
964
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
965
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
966 def _entries_walk(repo, includes, excludes, includeobsmarkers: bool):
50512
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
967 """emit a seris of files information useful to clone a repo
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
968
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
969 return (vfs-key, entry) iterator
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
970
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
971 Where `entry` is StoreEntry. (used even for cache entries)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
972 """
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
973 assert repo._currentlock(repo._lockref) is not None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
974
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
975 matcher = None
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
976 if includes or excludes:
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
977 matcher = narrowspec.match(repo.root, includes, excludes)
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
978
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
979 phase = not repo.publishing()
51547
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
980 # Python is getting crazy at all the small container we creates, disabling
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
981 # the gc while we do so helps performance a lot.
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
982 with util.nogc():
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
983 entries = _walkstreamfiles(
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
984 repo,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
985 matcher,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
986 phase=phase,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
987 obsolescence=includeobsmarkers,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
988 )
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
989 for entry in entries:
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
990 yield (_srcstore, entry)
50512
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
991
51547
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
992 for name in cacheutil.cachetocopy(repo):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
993 if repo.cachevfs.exists(name):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51545
diff changeset
994 # not really a StoreEntry, but close enough
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
995 yield (_srccache, CacheEntry(entry_path=name))
50513
43ed1f12b00a stream-clone: yield cache entry in `_entries_walk` too
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50512
diff changeset
996
50512
06d580b8f432 stream-clone: introduce a _entries_walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
997
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
998 def generatev2(repo, includes, excludes, includeobsmarkers: bool):
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
999 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1000
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1001 the data stream consists the following entries:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1002 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1003 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1004 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1005 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1006 5) N bytes containing the file data
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1007
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1008 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1009 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1010
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1011 with repo.lock():
47444
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1012 repo.ui.debug(b'scanning\n')
40339
f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40338
diff changeset
1013
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1014 entries = _entries_walk(
47444
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1015 repo,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1016 includes=includes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1017 excludes=excludes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1018 includeobsmarkers=includeobsmarkers,
2f4ca4807033 streamingclone: extract the scanning part from the generation part
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1019 )
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1020 chunks = _emit2(repo, entries)
35765
56c30b31afbe streamclone: add support for cloning non append-only file
Boris Feld <boris.feld@octobus.net>
parents: 35757
diff changeset
1021 first = next(chunks)
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1022 file_count, total_file_size = first
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1023 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1024 _test_sync_point_walk_4(repo)
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1025
50632
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50631
diff changeset
1026 return file_count, total_file_size, chunks
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1027
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1028
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1029 def generatev3(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1030 repo, includes, excludes, includeobsmarkers: bool
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1031 ) -> Iterator[bytes | None]:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1032 """Emit content for version 3 of a streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1033
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1034 the data stream consists the following:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1035 1) A varint E containing the number of entries (can be 0), then E entries follow
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1036 2) For each entry:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1037 2.1) The number of files in this entry (can be 0, but typically 1 or 2)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1038 2.2) For each file:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1039 2.2.1) A char representing the file destination (eg: store or cache)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1040 2.2.2) A varint N containing the length of the filename
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1041 2.2.3) A varint M containing the length of file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1042 2.2.4) N bytes containing the filename (the internal, store-agnostic form)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1043 2.2.5) M bytes containing the file data
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1044
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1045 Returns the data iterator.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1046
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1047 XXX This format is experimental and subject to change. Here is a
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1048 XXX non-exhaustive list of things this format could do or change:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1049
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1050 - making it easier to write files in parallel
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1051 - holding the lock for a shorter time
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1052 - improving progress information
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1053 - ways to adjust the number of expected entries/files ?
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1054 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1055
51548
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1056 # Python is getting crazy at all the small container we creates while
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1057 # considering the files to preserve, disabling the gc while we do so helps
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1058 # performance a lot.
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51547
diff changeset
1059 with repo.lock(), util.nogc():
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1060 repo.ui.debug(b'scanning\n')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1061
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1062 entries = _entries_walk(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1063 repo,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1064 includes=includes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1065 excludes=excludes,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1066 includeobsmarkers=includeobsmarkers,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1067 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1068 chunks = _emit3(repo, list(entries))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1069 first = next(chunks)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1070 assert first is None
52358
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1071 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
1072 _test_sync_point_walk_4(repo)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1073
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1074 return chunks
50521
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50520
diff changeset
1075
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50520
diff changeset
1076
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1077 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1078 def nested(*ctxs):
39757
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1079 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1080 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1081 with this:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1082 if rest:
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1083 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1084 yield
97f2992c26f6 streamclone: reimplement nested context manager
Augie Fackler <augie@google.com>
parents: 39700
diff changeset
1085 else:
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1086 yield
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1087
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1088
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1089 class V2Report:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1090 """a small class to track the data we saw within the stream"""
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1091
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1092 def __init__(self):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1093 self.byte_count = 0
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1094
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1095
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1096 def consumev2(repo, fp, filecount: int, filesize: int) -> None:
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1097 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1098
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1099 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1100 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1101 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1102 with repo.lock():
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1103 repo.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1104 _(b'%d files to transfer, %s of data\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1105 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1106 )
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1107 progress = repo.ui.makeprogress(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1108 _(b'clone'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1109 total=filesize,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1110 unit=_(b'bytes'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1111 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1112 start = util.timer()
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1113 report = V2Report()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1114
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1115 vfsmap = _makemap(repo)
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1116 # we keep repo.vfs out of the on purpose, ther are too many danger
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1117 # there (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1118 #
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1119 # this assert is duplicated (from _makemap) as author might think this
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1120 # is fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1121 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1122 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
1123 'repo.vfs must not be added to vfsmap for security reasons'
47502
65c519661991 stream: double check that self.vfs is *not* in the vfsmap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
1124 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1125
52913
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
1126 cpu_profile = scmutil.get_resource_profile(repo.ui, b'cpu')
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1127 mem_profile = scmutil.get_resource_profile(repo.ui, b'memory')
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1128 threaded = repo.ui.configbool(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1129 b"worker", b"parallel-stream-bundle-processing"
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1130 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1131 num_writer = repo.ui.configint(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1132 b"worker",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1133 b"parallel-stream-bundle-processing.num-writer",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1134 )
52913
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
1135 if num_writer <= 0:
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52912
diff changeset
1136 num_writer = DEFAULT_NUM_WRITER[cpu_profile]
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1137 memory_target = repo.ui.configbytes(
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1138 b"worker",
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1139 b"parallel-stream-bundle-processing.memory-target",
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1140 )
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1141 if memory_target < 0:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1142 memory_target = None
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1143 elif memory_target == 0:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1144 memory_target = DEFAULT_MEMORY_TARGET[mem_profile]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1145 with repo.transaction(b'clone'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1146 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767
5f5fb279fd39 streamclone: also stream caches to the client
Boris Feld <boris.feld@octobus.net>
parents: 35765
diff changeset
1147 with nested(*ctxs):
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1148 workers = []
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1149 info_queue = None
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1150 data_queue = None
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1151 mark_used = None
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1152 try:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1153 if not threaded:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1154 fc = _FileChunker
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1155 raw_data = fp
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1156 else:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1157 fc = _ThreadSafeFileChunker
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1158 data_queue = _DataQueue(memory_target=memory_target)
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1159 mark_used = data_queue.mark_used
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1160 raw_data = util.chunkbuffer(data_queue)
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1161
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1162 w = threading.Thread(
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1163 target=data_queue.fill_from,
52917
9abf173a958b stream-clone-v2: use the pass through for threaded stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52914
diff changeset
1164 args=(fp,),
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1165 )
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1166 workers.append(w)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1167 w.start()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1168 files = _v2_parse_files(
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1169 repo,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1170 raw_data,
52929
5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52928
diff changeset
1171 vfsmap,
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1172 filecount,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1173 progress,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1174 report,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1175 file_chunker=fc,
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1176 mark_used=mark_used,
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1177 )
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1178 if not threaded:
52930
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52929
diff changeset
1179 _write_files(files)
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1180 else:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1181 info_queue = _FileInfoQueue(files)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1182
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1183 for __ in range(num_writer):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1184 w = threading.Thread(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1185 target=_write_files,
52930
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52929
diff changeset
1186 args=(info_queue,),
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1187 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1188 workers.append(w)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1189 w.start()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1190 info_queue.fill()
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1191 except: # re-raises
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1192 if data_queue is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1193 data_queue.abort()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1194 raise
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1195 finally:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1196 # shut down all the workers
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1197 if info_queue is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1198 # this is strictly speaking one too many worker for
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1199 # this queu, but closing too many is not a problem.
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1200 info_queue.close(len(workers))
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1201 for w in workers:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1202 w.join()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1203
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1204 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1205 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1206 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1207
38373
ef692614e601 progress: hide update(None) in a new complete() method
Martin von Zweigbergk <martinvonz@google.com>
parents: 38360
diff changeset
1208 progress.complete()
52703
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1209 # acknowledge the end of the bundle2 part, this help aligning
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1210 # sequential and parallel behavior.
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1211 remains = fp.read(1)
2e82bd50978c stream: acknowledge the end of the bundle part earlier
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52702
diff changeset
1212 assert not remains
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1213 _report_transferred(repo, start, filecount, report.byte_count)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1214
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1215
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1216 # iterator of chunk of bytes that constitute a file content.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1217 FileChunksT = Iterator[bytes]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1218 # Contains the information necessary to write stream file on disk
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1219 FileInfoT = Tuple[
52930
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52929
diff changeset
1220 bytes, # real fs path
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52929
diff changeset
1221 Optional[int], # permission to give to chmod
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1222 FileChunksT, # content
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1223 ]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1224
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1225
52922
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1226 class _Queue:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1227 """a reimplementation of queue.Queue which doesn't use thread.Condition"""
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1228
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1229 def __init__(self):
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1230 self._queue = collections.deque()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1231
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1232 # the "_lock" protect manipulation of the "_queue" deque
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1233 # the "_wait" is used to have the "get" thread waits for the
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1234 # "put" thread when the queue is empty.
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1235 #
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1236 # This is similar to the "threading.Condition", but without the absurd
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1237 # slowness of the stdlib implementation.
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1238 #
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1239 # the "_wait" is always released while holding the "_lock".
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1240 self._lock = threading.Lock()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1241 self._wait = threading.Lock()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1242
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1243 def put(self, item):
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1244 with self._lock:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1245 self._queue.append(item)
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1246 # if anyone is waiting on item, unblock it.
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1247 if self._wait.locked():
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1248 self._wait.release()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1249
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1250 def get(self):
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1251 with self._lock:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1252 while len(self._queue) == 0:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1253 # "arm" the waiting lock
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1254 self._wait.acquire(blocking=False)
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1255 # release the lock to let other touch the queue
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1256 # (especially the put call we wait on)
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1257 self._lock.release()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1258 # wait for for a `put` call to release the lock
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1259 self._wait.acquire()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1260 # grab the lock to look at a possible available value
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1261 self._lock.acquire()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1262 # disarm the lock if necessary.
52923
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1263 #
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1264 # If the queue only constains one item, keep the _wait lock
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1265 # armed, as there is no need to wake another waiter anyway.
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52922
diff changeset
1266 if self._wait.locked() and len(self._queue) > 1:
52922
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1267 self._wait.release()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1268 return self._queue.popleft()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1269
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1270
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1271 class _DataQueue:
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1272 """A queue passing data from the bundle stream to other thread
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1273
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1274 It has a "target_memory" optional parameter to avoid buffering too much
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1275 information. The implementation is not exact and the memory target might be
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1276 exceed for a time in some situation.
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1277 """
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1278
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1279 def __init__(self, memory_target=None):
52922
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1280 self._q = _Queue()
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1281 self._abort = False
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1282 self._memory_target = memory_target
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1283 if self._memory_target is not None and self._memory_target <= 0:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1284 raise error.ProgrammingError("memory target should be > 0")
52921
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1285
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1286 # the "_lock" protect manipulation of the _current_used" variable
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1287 # the "_wait" is used to have the "reading" thread waits for the
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1288 # "using" thread when the buffer is full.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1289 #
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1290 # This is similar to the "threading.Condition", but without the absurd
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1291 # slowness of the stdlib implementation.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1292 #
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1293 # the "_wait" is always released while holding the "_lock".
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1294 self._lock = threading.Lock()
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1295 self._wait = threading.Lock()
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1296 # only the stream reader touch this, it is find to touch without the lock
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1297 self._current_read = 0
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1298 # do not touch this without the lock
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1299 self._current_used = 0
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1300
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1301 def _has_free_space(self):
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1302 """True if more data can be read without further exceeding memory target
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1303
52921
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1304 Must be called under the lock.
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1305 """
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1306 if self._memory_target is None:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1307 # Ideally we should not even get into the locking business in that
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1308 # case, but we keep the implementation simple for now.
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1309 return True
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1310 return (self._current_read - self._current_used) < self._memory_target
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1311
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1312 def mark_used(self, offset):
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1313 """Notify we have used the buffer up to "offset"
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1314
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1315 This is meant to be used from another thread than the one filler the queue.
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1316 """
52921
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1317 with self._lock:
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1318 if offset > self._current_used:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1319 self._current_used = offset
52921
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1320 # If the reader is waiting for room, unblock it.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1321 if self._wait.locked() and self._has_free_space():
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1322 self._wait.release()
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1323
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1324 def fill_from(self, data):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1325 """fill the data queue from a bundle2 part object
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1326
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1327 This is meant to be called by the data reading thread
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1328 """
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1329 q = self._q
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1330 try:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1331 for item in data:
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1332 self._current_read += len(item)
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1333 q.put(item)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1334 if self._abort:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1335 break
52921
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1336 with self._lock:
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1337 while not self._has_free_space():
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1338 # make sure the _wait lock is locked
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1339 # this is done under lock, so there case be no race with the release logic
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1340 self._wait.acquire(blocking=False)
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1341 self._lock.release()
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1342 # acquiring the lock will block until some other thread release it.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1343 self._wait.acquire()
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1344 # lets dive into the locked section again
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1345 self._lock.acquire()
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1346 # make sure we release the lock we just grabed if
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1347 # needed.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1348 if self._wait.locked():
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1349 self._wait.release()
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1350 finally:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1351 q.put(None)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1352
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1353 def __iter__(self):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1354 """Iterate over the bundle chunkgs
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1355
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1356 This is meant to be called by the data parsing thread."""
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1357 q = self._q
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1358 while (i := q.get()) is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1359 yield i
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1360 if self._abort:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1361 break
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1362
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1363 def abort(self):
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1364 """stop the data-reading thread and interrupt the comsuming iteration
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1365
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1366 This is meant to be called on errors.
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1367 """
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1368 self._abort = True
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1369 self._q.put(None)
52921
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1370 with self._lock:
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1371 # make sure we unstuck the reader thread.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1372 if self._wait.locked():
363914ba328d stream-clone-v2: no longer use the stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52920
diff changeset
1373 self._wait.release()
52914
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1374
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52913
diff changeset
1375
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1376 class _FileInfoQueue:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1377 """A thread-safe queue to passer parsed file information to the writers"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1378
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1379 def __init__(self, info: Iterable[FileInfoT]):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1380 self._info = info
52922
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1381 self._q = _Queue()
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1382
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1383 def fill(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1384 """iterate over the parsed information to file the queue
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1385
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1386 This is meant to be call from the thread parsing the stream information.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1387 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1388 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1389 for i in self._info:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1390 q.put(i)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1391
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1392 def close(self, number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1393 """signal all the workers that we no longer have any file info coming
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1394
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1395 Called from the thread parsing the stream information (and/or the main
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1396 thread if different).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1397 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1398 for __ in range(number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1399 self._q.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1400
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1401 def __iter__(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1402 """iterate over the available file info
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1403
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1404 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1405 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1406 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1407 while (i := q.get()) is not None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1408 yield i
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1409
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1410
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1411 class _FileChunker:
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1412 """yield the chunk that constitute a file
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1413
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1414 This class exists as the counterpart of the threaded version and
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1415 would not be very useful on its own.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1416 """
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1417
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1418 def __init__(
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1419 self,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1420 fp: bundle2mod.unbundlepart,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1421 data_len: int,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1422 progress: scmutil.progress,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1423 report: V2Report,
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1424 mark_used: Optional[Callable[[int], None]] = None,
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1425 ):
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1426 self.report = report
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1427 self.progress = progress
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1428 self._chunks = util.filechunkiter(fp, limit=data_len)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1429
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1430 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1431 """Do nothing in non-threading context"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1432
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1433 def __iter__(self) -> FileChunksT:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1434 for chunk in self._chunks:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1435 self.report.byte_count += len(chunk)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1436 self.progress.increment(step=len(chunk))
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1437 yield chunk
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1438
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1439
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1440 class _ThreadSafeFileChunker(_FileChunker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1441 """yield the chunk that constitute a file
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1442
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1443 Make sure you call the "fill" function in the main thread to read the
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1444 right data at the right time.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1445 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1446
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1447 def __init__(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1448 self,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1449 fp: bundle2mod.unbundlepart,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1450 data_len: int,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1451 progress: scmutil.progress,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1452 report: V2Report,
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1453 mark_used: Optional[Callable[[int], None]] = None,
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1454 ):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1455 super().__init__(fp, data_len, progress, report)
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1456 self._fp = fp
52922
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52921
diff changeset
1457 self._queue = _Queue()
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1458 self._mark_used = mark_used
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1459
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1460 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1461 """fill the file chunker queue with data read from the stream
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1462
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1463 This is meant to be called from the thread parsing information (and
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1464 consuming the stream data).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1465 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1466 try:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1467 for chunk in super().__iter__():
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1468 offset = self._fp.tell()
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1469 self._queue.put((chunk, offset))
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1470 finally:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1471 self._queue.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1472
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1473 def __iter__(self) -> FileChunksT:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1474 """Iterate over all the file chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1475
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1476 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1477 """
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1478 while (info := self._queue.get()) is not None:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1479 chunk, offset = info
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1480 if self._mark_used is not None:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1481 self._mark_used(offset)
52912
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1482 yield chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1483
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52911
diff changeset
1484
52925
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1485 def _trivial_file(
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1486 chunk: bytes,
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1487 mark_used: Optional[Callable[[int], None]],
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1488 offset: int,
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1489 ) -> FileChunksT:
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1490 """used for single chunk file,"""
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1491 if mark_used is not None:
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1492 mark_used(offset)
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1493 yield chunk
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1494
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1495
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1496 def _v2_parse_files(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1497 repo,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1498 fp: bundle2mod.unbundlepart,
52929
5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52928
diff changeset
1499 vfs_map,
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1500 file_count: int,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1501 progress: scmutil.progress,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1502 report: V2Report,
52911
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52910
diff changeset
1503 file_chunker: Type[_FileChunker] = _FileChunker,
52920
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52917
diff changeset
1504 mark_used: Optional[Callable[[int], None]] = None,
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1505 ) -> Iterator[FileInfoT]:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1506 """do the "stream-parsing" part of stream v2
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1507
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1508 The parsed information are yield result for consumption by the "writer"
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1509 """
52929
5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52928
diff changeset
1510 known_dirs = set() # set of directory that we know to exists
52909
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1511 progress.update(0)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1512 for i in range(file_count):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1513 src = util.readexactly(fp, 1)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1514 namelen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1515 datalen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1516
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1517 name = util.readexactly(fp, namelen)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1518
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1519 if repo.ui.debugflag:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1520 repo.ui.debug(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1521 b'adding [%s] %s (%s)\n' % (src, name, util.bytecount(datalen))
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52703
diff changeset
1522 )
52929
5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52928
diff changeset
1523 vfs = vfs_map[src]
52930
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52929
diff changeset
1524 path, mode = vfs.prepare_streamed_file(name, known_dirs)
52925
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1525 if datalen <= util.DEFAULT_FILE_CHUNK:
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1526 c = fp.read(datalen)
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1527 offset = fp.tell()
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1528 report.byte_count += len(c)
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1529 progress.increment(step=len(c))
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1530 chunks = _trivial_file(c, mark_used, offset)
52930
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52929
diff changeset
1531 yield (path, mode, iter(chunks))
52925
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1532 else:
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1533 chunks = file_chunker(
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1534 fp,
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1535 datalen,
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1536 progress,
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1537 report,
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1538 mark_used=mark_used,
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1539 )
52930
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52929
diff changeset
1540 yield (path, mode, iter(chunks))
52925
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1541 # make sure we read all the chunk before moving to the next file
119cddd02b8c stream-clone-v2: simplify the handling of small file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52923
diff changeset
1542 chunks.fill()
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1543
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1544
52930
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52929
diff changeset
1545 def _write_files(info: Iterable[FileInfoT]):
52910
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1546 """write files from parsed data"""
52971
c636f2835c95 stream-bundle: make sure to open the file in binary mode
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52970
diff changeset
1547 io_flags = os.O_WRONLY | os.O_CREAT
c636f2835c95 stream-bundle: make sure to open the file in binary mode
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52970
diff changeset
1548 if pycompat.iswindows:
c636f2835c95 stream-bundle: make sure to open the file in binary mode
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52970
diff changeset
1549 io_flags |= os.O_BINARY
52930
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52929
diff changeset
1550 for path, mode, data in info:
52970
e4ff37b5317c stream-bundle: simple use of `mode` argument of `os.open`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52931
diff changeset
1551 if mode is None:
52971
c636f2835c95 stream-bundle: make sure to open the file in binary mode
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52970
diff changeset
1552 fd = os.open(path, io_flags)
52970
e4ff37b5317c stream-bundle: simple use of `mode` argument of `os.open`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52931
diff changeset
1553 else:
52971
c636f2835c95 stream-bundle: make sure to open the file in binary mode
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52970
diff changeset
1554 fd = os.open(path, io_flags, mode=mode)
52931
c124308e3cd4 stream-clone-v2: directly use the os module for file operation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52930
diff changeset
1555 try:
52910
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1556 for chunk in data:
52931
c124308e3cd4 stream-clone-v2: directly use the os module for file operation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52930
diff changeset
1557 written = os.write(fd, chunk)
52928
7fc882f7fada stream-clone-v2: disable buffering when writing the files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52926
diff changeset
1558 # write missing pieces if the write was interrupted
7fc882f7fada stream-clone-v2: disable buffering when writing the files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52926
diff changeset
1559 while written < len(chunk):
52931
c124308e3cd4 stream-clone-v2: directly use the os module for file operation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52930
diff changeset
1560 written = os.write(fd, chunk[written:])
c124308e3cd4 stream-clone-v2: directly use the os module for file operation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52930
diff changeset
1561 finally:
c124308e3cd4 stream-clone-v2: directly use the os module for file operation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52930
diff changeset
1562 os.close(fd)
52910
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1563
70306aefa52b stream-clone-v2: extract the file writing code in a function
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52909
diff changeset
1564
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1565 def consumev3(repo, fp) -> None:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1566 """Apply the contents from a version 3 streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1567
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1568 Data is read from an object that only needs to provide a ``read(size)``
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1569 method.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1570 """
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1571 with repo.lock():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1572 start = util.timer()
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1573
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1574 entrycount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1575 repo.ui.status(_(b'%d entries to transfer\n') % (entrycount))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1576
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1577 progress = repo.ui.makeprogress(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1578 _(b'clone'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1579 total=entrycount,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1580 unit=_(b'entries'),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1581 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1582 progress.update(0)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1583 bytes_transferred = 0
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1584
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1585 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1586 # we keep repo.vfs out of the on purpose, there are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1587 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1588 #
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1589 # this assert is duplicated (from _makemap) as authors might think this
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1590 # is fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1591 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1592 raise error.ProgrammingError(
52696
10e7adbffa8c streamclone: unbyteify string args to builtin Error classes
Matt Harbison <matt_harbison@yahoo.com>
parents: 52695
diff changeset
1593 'repo.vfs must not be added to vfsmap for security reasons'
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1594 )
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
1595 total_file_count = 0
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1596 with repo.transaction(b'clone'):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1597 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1598 with nested(*ctxs):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1599 for i in range(entrycount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1600 filecount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1601 if filecount == 0:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1602 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1603 repo.ui.debug(b'entry with no files [%d]\n' % (i))
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
1604 total_file_count += filecount
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1605 for i in range(filecount):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1606 src = util.readexactly(fp, 1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1607 vfs = vfsmap[src]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1608 namelen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1609 datalen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1610
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1611 name = util.readexactly(fp, namelen)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1612
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1613 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1614 msg = b'adding [%s] %s (%s)\n'
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1615 msg %= (src, name, util.bytecount(datalen))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1616 repo.ui.debug(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1617 bytes_transferred += datalen
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1618
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1619 with vfs(name, b'w') as ofp:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1620 for chunk in util.filechunkiter(fp, limit=datalen):
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1621 ofp.write(chunk)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1622 progress.increment(step=1)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1623
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1624 # force @filecache properties to be reloaded from
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1625 # streamclone-ed file at next access
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1626 repo.invalidate(clearfilecache=True)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1627
52701
6feb3b3029b5 stream: consistently close progress before reporting time
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52700
diff changeset
1628 progress.complete()
52702
ca17d31624ac stream: report number of file written then information is available
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52701
diff changeset
1629 _report_transferred(repo, start, total_file_count, bytes_transferred)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1630
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1631
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1632 def applybundlev2(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1633 repo, fp, filecount: int, filesize: int, requirements: Iterable[bytes]
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1634 ) -> None:
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
1635 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
1636
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1637 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1638 if missingreqs:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1639 raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43085
diff changeset
1640 _(b'unable to apply stream clone: unsupported format: %s')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
1641 % b', '.join(sorted(missingreqs))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1642 )
35756
cfdccd560b66 streamclone: define first iteration of version 2 of stream format
Boris Feld <boris.feld@octobus.net>
parents: 35491
diff changeset
1643
52926
34fa51c25112 stream-clone-v2: disable the garbage collector during consumption
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52925
diff changeset
1644 with util.nogc():
34fa51c25112 stream-clone-v2: disable the garbage collector during consumption
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52925
diff changeset
1645 consumev2(repo, fp, filecount, filesize)
35804
2d3e486d09d0 streamclone: move requirement update into consumev2
Boris Feld <boris.feld@octobus.net>
parents: 35803
diff changeset
1646
48596
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
1647 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
1648 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47878
diff changeset
1649 requirements,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1650 )
39700
b10d145837bc localrepo: extract resolving of opener options to standalone functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 38783
diff changeset
1651 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1652 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42813
diff changeset
1653 )
45106
a03c177a4679 scmutil: add writereporequirements() and route requires writing through it
Pulkit Goyal <7895pulkit@gmail.com>
parents: 43117
diff changeset
1654 scmutil.writereporequirements(repo)
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48681
diff changeset
1655 nodemap.post_stream_cleanup(repo)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1656
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1657
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1658 def applybundlev3(repo, fp, requirements: Iterable[bytes]) -> None:
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1659 from . import localrepo
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1660
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1661 missingreqs = [r for r in requirements if r not in repo.supported]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1662 if missingreqs:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1663 msg = _(b'unable to apply stream clone: unsupported format: %s')
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1664 msg %= b', '.join(sorted(missingreqs))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1665 raise error.Abort(msg)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1666
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1667 consumev3(repo, fp)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1668
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1669 repo.requirements = new_stream_clone_requirements(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1670 repo.requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1671 requirements,
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1672 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1673 repo.svfs.options = localrepo.resolvestorevfsoptions(
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1674 repo.ui, repo.requirements, repo.features
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1675 )
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1676 scmutil.writereporequirements(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1677 nodemap.post_stream_cleanup(repo)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1678
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50663
diff changeset
1679
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1680 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress) -> bool:
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1681 hardlink = [True]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1682
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1683 def copy_used():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1684 hardlink[0] = False
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1685 progress.topic = _(b'copying')
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1686
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1687 for k, path, optional in entries:
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1688 src_vfs = src_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1689 dst_vfs = dst_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1690 src_path = src_vfs.join(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1691 dst_path = dst_vfs.join(path)
47871
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1692 # We cannot use dirname and makedirs of dst_vfs here because the store
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1693 # encoding confuses them. See issue 6581 for details.
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1694 dirname = os.path.dirname(dst_path)
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1695 if not os.path.exists(dirname):
132525ead0db clone: properly create target directories during local clone (issue6581)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47788
diff changeset
1696 util.makedirs(dirname)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1697 dst_vfs.register_file(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1698 # XXX we could use the #nb_bytes argument.
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1699 try:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1700 util.copyfile(
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1701 src_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1702 dst_path,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1703 hardlink=hardlink[0],
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1704 no_hardlink_cb=copy_used,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1705 check_fs_hardlink=False,
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1706 )
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1707 except FileNotFoundError:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1708 if not optional:
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1709 raise
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1710 progress.increment()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1711 return hardlink[0]
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1712
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1713
52695
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52691
diff changeset
1714 def local_copy(src_repo, dest_repo) -> None:
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1715 """copy all content from one local repository to another
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1716
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1717 This is useful for local clone"""
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1718 src_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1719 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1720 for r in src_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1721 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1722 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1723 dest_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1724 r
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1725 for r in dest_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1726 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1727 }
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1728 assert src_store_requirements == dest_store_requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1729
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1730 with dest_repo.lock():
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1731 with src_repo.lock():
47448
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1732 # bookmark is not integrated to the streaming as it might use the
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1733 # `repo.vfs` and they are too many sentitive data accessible
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1734 # through `repo.vfs` to expose it to streaming clone.
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1735 src_book_vfs = bookmarks.bookmarksvfs(src_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1736 srcbookmarks = src_book_vfs.join(b'bookmarks')
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1737 bm_count = 0
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1738 if os.path.exists(srcbookmarks):
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1739 bm_count = 1
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1740
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1741 entries = _entries_walk(
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1742 src_repo,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1743 includes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1744 excludes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1745 includeobsmarkers=True,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1746 )
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1747 entries = list(entries)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1748 src_vfs_map = _makemap(src_repo)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1749 dest_vfs_map = _makemap(dest_repo)
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1750 total_files = sum(len(e[1].files()) for e in entries) + bm_count
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1751 progress = src_repo.ui.makeprogress(
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1752 topic=_(b'linking'),
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1753 total=total_files,
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1754 unit=_(b'files'),
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1755 )
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1756 # copy files
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1757 #
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1758 # We could copy the full file while the source repository is locked
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1759 # and the other one without the lock. However, in the linking case,
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1760 # this would also requires checks that nobody is appending any data
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1761 # to the files while we do the clone, so this is not done yet. We
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1762 # could do this blindly when copying files.
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1763 files = [
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52358
diff changeset
1764 (vfs_key, f.unencoded_path, f.optional)
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1765 for vfs_key, e in entries
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1766 for f in e.files()
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1767 ]
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1768 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1769
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1770 # copy bookmarks over
47448
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1771 if bm_count:
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1772 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
d370256636fe clone: also report the bookmark file as copied
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47447
diff changeset
1773 dstbookmarks = dst_book_vfs.join(b'bookmarks')
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1774 util.copyfile(srcbookmarks, dstbookmarks)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1775 progress.complete()
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1776 if hardlink:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1777 msg = b'linked %d files\n'
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1778 else:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1779 msg = b'copied %d files\n'
50630
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50627
diff changeset
1780 src_repo.ui.debug(msg % total_files)
47447
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1781
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1782 with dest_repo.transaction(b"localclone") as tr:
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1783 dest_repo.store.write(tr)
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1784
377d8fc20e34 clone: reuse the stream clone logic for local clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47444
diff changeset
1785 # clean up transaction file as they do not make sense
50288
d89eecf9605e undo-files: no longer pass the `repo` to `cleanup_undo_files`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50286
diff changeset
1786 transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)