mercurial-scm/hg: mercurial/streamclone.py annotate

annotate mercurial/streamclone.py @ 53042:cdd7bf612c7b stable tip

bundle-spec: properly format boolean parameter (issue6960) This was breaking automatic clone bundle generation. This changeset fixes it and add a test to catch it in the future.

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Tue, 11 Mar 2025 02:29:42 +0100
parents	e705fec4a03f
children

rev	line source
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
51859 f4733654f144 typing: add `from __future__ import annotations` to most files Matt Harbison <matt_harbison@yahoo.com> parents: 51699 diff changeset	8 from __future__ import annotations
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	10 import collections
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	11 import contextlib
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	12 import errno
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	13 import os
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	14 import struct
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	15 import threading
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	16
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	17 from typing import (
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	18 Callable,
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	19 Iterable,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	20 Iterator,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	21 Optional,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	22 Set,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	23 Tuple,
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	24 Type,
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	25 )
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	26
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	27 from .i18n import _
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	28 from .interfaces import repository
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	29 from . import (
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	30 bookmarks,
50520 a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	31 bundle2 as bundle2mod,
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	32 cacheutil,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	33 error,
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	34 narrowspec,
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	35 phases,
38164 aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path Yuya Nishihara <yuya@tcha.org> parents: 37638 diff changeset	36 pycompat,
46626 ee91966aec0f requirements: add constant for revlog v1 requirement Rapha?l Gom?s <rgomes@octobus.net> parents: 45106 diff changeset	37 requirements as requirementsmod,
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	38 scmutil,
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	39 store,
50286 3d0b5760851c undo-files: move the undo cleanup code in the transaction module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50279 diff changeset	40 transaction,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	41 util,
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	42 )
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	43 from .revlogutils import (
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	44 nodemap,
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	45 )
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	46
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	47
52913 58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	48 # Number arbitrarily picked, feel free to change them (but the LOW one)
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	49 #
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	50 # update the configuration documentation if you touch this.
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	51 DEFAULT_NUM_WRITER = {
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	52 scmutil.RESOURCE_LOW: 1,
52997 e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	53 scmutil.RESOURCE_MEDIUM: 4,
e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	54 scmutil.RESOURCE_HIGH: 8,
52913 58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	55 }
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	56
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	57
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	58 # Number arbitrarily picked, feel free to adjust them. Do update the
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	59 # documentation if you do so
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	60 DEFAULT_MEMORY_TARGET = {
52997 e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	61 scmutil.RESOURCE_LOW: 50 * (2**20), # 100 MB
e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	62 scmutil.RESOURCE_MEDIUM: 500 * 2**20, # 500 MB
e75ed9ae5fb9 stream-clone: adjust default value for worker and memory Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52996 diff changeset	63 scmutil.RESOURCE_HIGH: 2 * 2**30, # 2 GB
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	64 }
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	65
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	66
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	67 def new_stream_clone_requirements(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	68 default_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	69 streamed_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	70 ) -> Set[bytes]:
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	71 """determine the final set of requirement for a new stream clone
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	72
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	73 this method combine the "default" requirements that a new repository would
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	74 use with the constaint we get from the stream clone content. We keep local
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	75 configuration choice when possible.
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	76 """
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	77 requirements = set(default_requirements)
53028 89ab2459f62a stream-bundle: properly process requirements Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	78 requirements & requirementsmod.STREAM_IGNORABLE_REQUIREMENTS
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	79 requirements.update(streamed_requirements)
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	80 return requirements
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	81
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	82
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	83 def streamed_requirements(repo) -> Set[bytes]:
48597 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	84 """the set of requirement the new clone will have to support
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	85
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	86 This is used for advertising the stream options and to generate the actual
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	87 stream content."""
48601 baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	88 requiredformats = (
53028 89ab2459f62a stream-bundle: properly process requirements Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	89 repo.requirements - requirementsmod.STREAM_IGNORABLE_REQUIREMENTS
48601 baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48600 diff changeset	90 )
48597 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	91 return requiredformats
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	92
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	93
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	94 def canperformstreamclone(pullop, bundle2: bool = False):
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	95 """Whether it is possible to perform a streaming clone as part of pull.
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	96
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	97 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	98 bundle2 and only through bundle2.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	99
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	100 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	101 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	102 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	103 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	104 """
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	105 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	106 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	107
50519 58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	108 # should we consider streaming clone at all ?
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	109 streamrequested = pullop.streamclonerequested
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	110 # If we don't have a preference, let the server decide for us. This
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	111 # likely only comes into play in LANs.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	112 if streamrequested is None:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	113 # The server can advertise whether to prefer streaming clone.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	114 streamrequested = remote.capable(b'stream-preferred')
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	115 if not streamrequested:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	116 return False, None
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50518 diff changeset	117
50517 0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	118 # Streaming clone only works on an empty destination repository
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	119 if len(repo):
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	120 return False, None
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50513 diff changeset	121
50518 f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	122 # Streaming clone only works if all data is being requested.
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	123 if pullop.heads:
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	124 return False, None
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50517 diff changeset	125
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	126 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	127 if pullop.canusebundle2:
50520 a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	128 local_caps = bundle2mod.getrepocaps(repo, role=b'client')
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	129 local_supported = set(local_caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	130 remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50519 diff changeset	131 bundle2supported = bool(local_supported & remote_supported)
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	132 # else
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	133 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	134 # the versions we support. Fall back and possibly allow legacy.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	135
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	136 # Ensures legacy code path uses available bundle2.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	137 if bundle2supported and not bundle2:
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	138 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	139 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	140 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	141 return False, None
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	142
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	143 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	144 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	145 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	146 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	147 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	148 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	149 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	150 requirements = set()
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	151 if remote.capable(b'stream'):
46626 ee91966aec0f requirements: add constant for revlog v1 requirement Rapha?l Gom?s <rgomes@octobus.net> parents: 45106 diff changeset	152 requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	153 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	154 streamreqs = remote.capable(b'streamreqs')
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	155 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	156 if not streamreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	157 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	158 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	159 b'warning: stream clone requested but server has them '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	160 b'disabled\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	161 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	162 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	163 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	164
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	165 streamreqs = set(streamreqs.split(b','))
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	166 # Server requires something we don't support. Bail.
48670 6d2ddea0721a stream-clone: filter possible missing requirements using all supported one Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	167 missingreqs = streamreqs - repo.supported
32259 076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30975 diff changeset	168 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30975 diff changeset	169 pullop.repo.ui.warn(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	170 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	171 b'warning: stream clone requested but client is missing '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	172 b'requirements: %s\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	173 )
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	174 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	175 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	176 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	177 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	178 b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	179 b'for more information)\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	180 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	181 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	182 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	183 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	184
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	185 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	186
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	187
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	188 def maybeperformlegacystreamclone(pullop) -> None:
26462 3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	189 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	190
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	191 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	192 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	193
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	194 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	195 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	196 """
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	197 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	198
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	199 supported, requirements = canperformstreamclone(pullop)
26458 362793295640 streamclone: refactor maybeperformstreamclone to take a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26447 diff changeset	200
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	201 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	202 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	203
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	204 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	205 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	206
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	207 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	208 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	209 rbranchmap = None
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	210 if remote.capable(b'branchmap'):
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	211 with remote.commandexecutor() as e:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	212 rbranchmap = e.callcommand(b'branchmap', {}).result()
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	213
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	214 repo.ui.status(_(b'streaming all changes\n'))
26470 4b5647d9ee13 streamclone: move "streaming all changes" message location Gregory Szorc <gregory.szorc@gmail.com> parents: 26469 diff changeset	215
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	216 with remote.commandexecutor() as e:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	217 fp = e.callcommand(b'stream_out', {}).result()
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	218
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	219 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	220 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	221 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	222 # doesn't matter.
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	223 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	224 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	225 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	226 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	227 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	228 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	229 )
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	230 if resp == 1:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	231 raise error.Abort(_(b'operation forbidden by server'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	232 elif resp == 2:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	233 raise error.Abort(_(b'locking the remote repository failed'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	234 elif resp != 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	235 raise error.Abort(_(b'the server sent an unknown error code'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	236
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	237 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	238 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	239 filecount, bytecount = map(int, l.split(b' ', 1))
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	240 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	241 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	242 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	243 )
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	244
27850 49cfddbf54ba with: use context manager in maybeperformlegacystreamclone Bryan O'Sullivan <bryano@fb.com> parents: 27845 diff changeset	245 with repo.lock():
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	246 consumev1(repo, fp, filecount, bytecount)
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	247 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	248 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	249 requirements,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	250 )
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	251 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	252 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	253 )
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	254 scmutil.writereporequirements(repo)
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	255 nodemap.post_stream_cleanup(repo)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	256
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	257 if rbranchmap:
41615 328ca3b9e545 branchmap: encapsulate cache updating in the map itself Martijn Pieters <mj@octobus.net> parents: 40494 diff changeset	258 repo._branchcaches.replace(repo, rbranchmap)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	259
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	260 repo.invalidate()
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	261
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	262
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	263 def allowservergeneration(repo) -> bool:
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	264 """Whether streaming clones are allowed from the server."""
40028 51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	265 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	266 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39850 diff changeset	267
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	268 if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	269 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	270
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	271 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	272 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	273 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	274 if secret:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	275 return repo.ui.configbool(b'server', b'uncompressedallowsecret')
32744 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	276
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32259 diff changeset	277 return True
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	278
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	279
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	280 # This is it's own function so extensions can override it.
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	281 def _walkstreamfiles(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	282 repo, matcher=None, phase: bool = False, obsolescence: bool = False
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	283 ):
50511 0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50509 diff changeset	284 return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	285
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	286
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	287 def _report_transferred(
ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	288 repo, start_time: float, file_count: int, byte_count: int
ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	289 ):
ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	290 """common utility to report time it took to apply the stream bundle"""
52700 061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	291 elapsed = util.timer() - start_time
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	292 if elapsed <= 0:
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	293 elapsed = 0.001
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	294 m = _(b'stream-cloned %d files / %s in %.1f seconds (%s/sec)\n')
52700 061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	295 m %= (
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	296 file_count,
52700 061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	297 util.bytecount(byte_count),
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	298 elapsed,
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	299 util.bytecount(byte_count / elapsed),
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	300 )
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	301 repo.ui.status(m)
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	302
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52696 diff changeset	303
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	304 def generatev1(repo) -> tuple[int, int, Iterator[bytes]]:
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	305 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	306
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	307 This returns a 3-tuple of (file count, byte size, data iterator).
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	308
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	309 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	310 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	311 delimited by a null byte.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	312
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	313 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	314 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	315
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	316 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	317 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	318 for adding it.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	319
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	320 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	321 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	322 """
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	323 entries = []
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	324 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	325 # Get consistent snapshot of repo, lock during scan.
27845 7417e1c10253 with: use context manager in streamclone generatev1 Bryan O'Sullivan <bryano@fb.com> parents: 27794 diff changeset	326 with repo.lock():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	327 repo.ui.debug(b'scanning\n')
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	328 _test_sync_point_walk_1_2(repo)
50468 521fec115dad store: use a StoreEntry object instead of tuple for store files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50288 diff changeset	329 for entry in _walkstreamfiles(repo):
50469 9fdc28e21b68 store: introduce a EntryFile object to actually access file info Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50468 diff changeset	330 for f in entry.files():
50474 4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	331 file_size = f.file_size(repo.store.vfs)
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	332 if file_size:
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	333 entries.append((f.unencoded_path, file_size))
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50469 diff changeset	334 total_bytes += file_size
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	335 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	336 _test_sync_point_walk_4(repo)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	337
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	338 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	339 b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	340 )
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	341
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	342 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	343 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	344
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	345 def emitrevlogdata() -> Iterator[bytes]:
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	346 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	347 if debugflag:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	348 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	349 # partially encode name over the wire for backwards compat
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	350 yield b'%s\0%d\n' % (store.encodedir(name), size)
33411 50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	351 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	352 # trusted by the local repo) and expensive
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	353 with svfs(name, b'rb', auditpath=False) as fp:
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	354 if size <= 65536:
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	355 yield fp.read(size)
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	356 else:
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52640 diff changeset	357 yield from util.filechunkiter(fp, limit=size)
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	358
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	359 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	360
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	361
52691 279e217d6041 typing: lock in the new type annotations detected with the pyupgrade changes Matt Harbison <matt_harbison@yahoo.com> parents: 52644 diff changeset	362 def generatev1wireproto(repo) -> Iterator[bytes]:
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	363 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	364
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	365 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	366 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	367 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	368
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	369 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	370 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	371 a permissions error for the server process).
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	372 """
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	373 if not allowservergeneration(repo):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	374 yield b'1\n'
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	375 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	376
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	377 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	378 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	379 except error.LockError:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	380 yield b'2\n'
35491 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	381 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	382
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	383 # Indicates successful response.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	384 yield b'0\n'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	385 yield b'%d %d\n' % (filecount, bytecount)
52644 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52640 diff changeset	386 yield from it
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	387
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	388
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	389 def generatebundlev1(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	390 repo, compression: bytes = b'UN'
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	391 ) -> tuple[Set[bytes], Iterator[bytes]]:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	392 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	393
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	394 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	395 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	396
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	397 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	398 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	399
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	400 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	401 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	402
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	403 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	404 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	405 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	406 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	407
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	408 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	409 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	410
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	411 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	412 """
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	413 if compression != b'UN':
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	414 raise ValueError('we do not support the compression argument yet')
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	415
48597 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48596 diff changeset	416 requirements = streamed_requirements(repo)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	417 requires = b','.join(sorted(requirements))
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	418
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	419 def gen() -> Iterator[bytes]:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	420 yield b'HGS1'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	421 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	422
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	423 filecount, bytecount, it = generatev1(repo)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	424 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	425 _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	426 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	427
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	428 yield struct.pack(b'>QQ', filecount, bytecount)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	429 yield struct.pack(b'>H', len(requires) + 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	430 yield requires + b'\0'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	431
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	432 # This is where we'll add compression in the future.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	433 assert compression == b'UN'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	434
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	435 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	436 _(b'bundle'), total=bytecount, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	437 )
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	438 progress.update(0)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	439
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	440 for chunk in it:
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	441 progress.increment(step=len(chunk))
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	442 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	443
38373 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38360 diff changeset	444 progress.complete()
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	445
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	446 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	447
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	448
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	449 def consumev1(repo, fp, filecount: int, bytecount: int) -> None:
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	450 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	451
30332 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29919 diff changeset	452 This takes the output from "stream_out" and applies it to the specified
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	453 repository.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	454
30332 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29919 diff changeset	455 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29919 diff changeset	456 handled by this function.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	457 """
27859 f55a5ace8e69 with: use context manager in streamclone consumev1 Bryan O'Sullivan <bryano@fb.com> parents: 27850 diff changeset	458 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	459 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	460 _(b'%d files to transfer, %s of data\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	461 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	462 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	463 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	464 _(b'clone'), total=bytecount, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	465 )
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	466 progress.update(0)
30975 22fbca1d11ed mercurial: switch to util.timer for all interval timings Simon Farnsworth <simonfar@fb.com> parents: 30332 diff changeset	467 start = util.timer()
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	468
29919 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	469 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	470 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	471 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	472 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	473 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	474 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	475 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	476 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	477 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	478 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	479 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	480 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	481 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	482
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	483 total_file_count = 0
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	484 with repo.transaction(b'clone'):
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	485 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
49284 d44e3c45f0e4 py3: replace `pycompat.xrange` by `range` Manuel Jacob <me@manueljacob.de> parents: 49190 diff changeset	486 for i in range(filecount):
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	487 # XXX doesn't support '\n' or '\r' in filenames
50925 d718eddf01d9 safehasattr: drop usage in favor of hasattr Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50703 diff changeset	488 if hasattr(fp, 'readline'):
50663 60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	489 l = fp.readline()
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	490 else:
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	491 # inline clonebundles use a chunkbuffer, so no readline
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	492 # --> this should be small anyway, the first line
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	493 # only contains the size of the bundle
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	494 l_buf = []
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	495 while not (l_buf and l_buf[-1] == b'\n'):
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	496 l_buf.append(fp.read(1))
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50637 diff changeset	497 l = b''.join(l_buf)
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	498 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	499 name, size = l.split(b'\0', 1)
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	500 size = int(size)
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	501 except (ValueError, TypeError):
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	502 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	503 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	504 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	505 if repo.ui.debugflag:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	506 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	507 b'adding %s (%s)\n' % (name, util.bytecount(size))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	508 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	509 # for backwards compat, name was partially encoded
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	510 path = store.decodedir(name)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	511 with repo.svfs(path, b'w', backgroundclose=True) as ofp:
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	512 total_file_count += 1
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	513 for chunk in util.filechunkiter(fp, limit=size):
38349 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38164 diff changeset	514 progress.increment(step=len(chunk))
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	515 ofp.write(chunk)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	516
29919 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	517 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	518 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29917 diff changeset	519 repo.invalidate(clearfilecache=True)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	520
38373 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38360 diff changeset	521 progress.complete()
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	522 _report_transferred(repo, start, total_file_count, bytecount)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	523
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	524
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	525 def readbundle1header(fp) -> tuple[int, int, Set[bytes]]:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	526 compression = fp.read(2)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	527 if compression != b'UN':
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	528 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	529 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	530 b'only uncompressed stream clone bundles are '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	531 b'supported; got %s'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	532 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	533 % compression
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	534 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	535
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	536 filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	537 requireslen = struct.unpack(b'>H', fp.read(2))[0]
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	538 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	539
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	540 if not requires.endswith(b'\0'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	541 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	542 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	543 b'malformed stream clone bundle: '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	544 b'requirements not properly encoded'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	545 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	546 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	547
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	548 requirements = set(requires.rstrip(b'\0').split(b','))
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	549
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	550 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	551
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	552
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	553 def applybundlev1(repo, fp) -> None:
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	554 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	555
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	556 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	557 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	558 """
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	559 if len(repo):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	560 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	561 _(b'cannot apply stream clone bundle on non-empty repo')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	562 )
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	563
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	564 filecount, bytecount, requirements = readbundle1header(fp)
48670 6d2ddea0721a stream-clone: filter possible missing requirements using all supported one Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	565 missingreqs = requirements - repo.supported
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	566 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	567 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	568 _(b'unable to apply stream clone: unsupported format: %s')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	569 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	570 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	571
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	572 consumev1(repo, fp, filecount, bytecount)
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	573 nodemap.post_stream_cleanup(repo)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	574
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	575
48946 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48875 diff changeset	576 class streamcloneapplier:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	577 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	578
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	579 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	580 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	581 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	582
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	583 def __init__(self, fh) -> None:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	584 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	585
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	586 def apply(self, repo) -> None:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	587 return applybundlev1(repo, self._fh)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	588
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	589
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	590 # type of file to stream
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	591 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	592 _filefull = 1 # full snapshot file
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	593
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	594 # Source of the file
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	595 _srcstore = b's' # store (svfs)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	596 _srccache = b'c' # cache (cache)
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	597
51699 ca7bde5dbafb black: format the codebase with 23.3.0 Rapha?l Gom?s <rgomes@octobus.net> parents: 51696 diff changeset	598
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	599 # This is it's own function so extensions can override it.
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	600 def _walkstreamfullstorefiles(repo) -> list[bytes]:
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	601 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	602 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	603 if not repo.publishing():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	604 fnames.append(b'phaseroots')
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	605 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	606
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	607
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	608 def _filterfull(entry, copy, vfsmap):
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	609 """actually copy the snapshot files"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	610 src, name, ftype, data = entry
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	611 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	612 return entry
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	613 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	614
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	615
52012 e308439339e2 stream: rename TempCopyManager to VolatileManager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51859 diff changeset	616 class VolatileManager:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	617 """Manage temporary backups of volatile files during stream clone.
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	618
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	619 This class will keep open file handles for the volatile files, writing the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	620 smaller ones on disk if the number of open file handles grow too much.
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	621
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	622 This should be used as a Python context, the file handles and copies will
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	623 be discarded when exiting the context.
49190 4ff4e23de7df clone: use better names for temp files Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 48946 diff changeset	624
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	625 The preservation can be done by calling the object on the real path
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	626 (encoded full path).
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	627
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	628 Valid filehandles for any file should be retrieved by calling `open(path)`.
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	629 """
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	630
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	631 # arbitrarily picked as "it seemed fine" and much higher than the current
52184 e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	632 # usage. The Windows value of 2 is actually 1 file open at a time, due to
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	633 # the `flush_count = self.MAX_OPEN // 2` and `self.MAX_OPEN - 1` threshold
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	634 # for flushing to disk in __call__().
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52015 diff changeset	635 MAX_OPEN = 2 if pycompat.iswindows else 100
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	636
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	637 def __init__(self) -> None:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	638 self._counter = 0
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	639 self._volatile_fps = None
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	640 self._copies = None
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	641 self._dst_dir = None
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	642
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	643 def __enter__(self):
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	644 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	645 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	646 self._volatile_fps = {}
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	647 self._counter += 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	648 return self
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	649
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	650 def __exit__(self, args, *kwars):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	651 """discard all backups"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	652 self._counter -= 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	653 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	654 for _size, fp in self._volatile_fps.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	655 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	656 self._volatile_fps = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	657 if self._copies is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	658 for tmp in self._copies.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	659 util.tryunlink(tmp)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	660 util.tryrmdir(self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	661 self._copies = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	662 self._dst_dir = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	663 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	664 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	665 assert self._dst_dir is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	666
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	667 def _init_tmp_copies(self) -> None:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	668 """prepare a temporary directory to save volatile files
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	669
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	670 This will be used as backup if we have too many files open"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	671 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	672 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	673 assert self._dst_dir is None
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	674 self._copies = {}
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	675 self._dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	676
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	677 def _flush_some_on_disk(self) -> None:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	678 """move some of the open files to tempory files on disk"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	679 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	680 self._init_tmp_copies()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	681 flush_count = self.MAX_OPEN // 2
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	682 for src, (size, fp) in sorted(self._volatile_fps.items())[:flush_count]:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	683 prefix = os.path.basename(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	684 fd, dst = pycompat.mkstemp(prefix=prefix, dir=self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	685 self._copies[src] = dst
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	686 os.close(fd)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	687 # we no longer hardlink, but on the other hand we rarely do this,
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	688 # and we do it for the smallest file only and not at all in the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	689 # common case.
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	690 with open(dst, 'wb') as bck:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	691 fp.seek(0)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	692 bck.write(fp.read())
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	693 del self._volatile_fps[src]
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	694 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	695
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	696 def _keep_one(self, src: bytes) -> int:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	697 """preserve an open file handle for a given path"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	698 # store the file quickly to ensure we close it if any error happens
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	699 _, fp = self._volatile_fps[src] = (None, open(src, 'rb'))
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	700 fp.seek(0, os.SEEK_END)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	701 size = fp.tell()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	702 self._volatile_fps[src] = (size, fp)
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	703 return size
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	704
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	705 def __call__(self, src: bytes) -> None:
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	706 """preserve the volatile file at src"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	707 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	708 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	709 self._flush_some_on_disk()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	710 self._keep_one(src)
50627 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50521 diff changeset	711
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	712 def try_keep(self, src: bytes) -> Optional[int]:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	713 """record a volatile file and returns it size
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	714
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	715 return None if the file does not exists.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	716
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	717 Used for cache file that are not lock protected.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	718 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	719 assert 0 < self._counter
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	720 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	721 self._flush_some_on_disk()
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	722 try:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	723 return self._keep_one(src)
52640 24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError` Matt Harbison <matt_harbison@yahoo.com> parents: 52359 diff changeset	724 except OSError as err:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	725 if err.errno not in (errno.ENOENT, errno.EPERM):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	726 raise
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	727 return None
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	728
52013 0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52012 diff changeset	729 @contextlib.contextmanager
0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52012 diff changeset	730 def open(self, src):
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	731 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	732 entry = self._volatile_fps.get(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	733 if entry is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	734 _size, fp = entry
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	735 fp.seek(0)
52013 0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52012 diff changeset	736 yield fp
52015 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	737 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	738 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	739 actual_path = src
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	740 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	741 actual_path = self._copies.get(src, src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	742 with open(actual_path, 'rb') as fp:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52014 diff changeset	743 yield fp
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	744
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	745
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	746 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	747 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	748 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	749 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	750 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	751 }
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	752 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	753 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	754 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	755
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	756 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	757
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	758
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	759 def _emit2(repo, entries):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	760 """actually emit the stream bundle"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	761 vfsmap = _makemap(repo)
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	762 # we keep repo.vfs out of the on purpose, ther are too many danger there
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	763 # (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	764 #
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	765 # this assert is duplicated (from _makemap) as author might think this is
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	766 # fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	767 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	768 raise error.ProgrammingError(
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	769 'repo.vfs must not be added to vfsmap for security reasons'
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	770 )
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	771
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	772 # translate the vfs one
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	773 entries = [(vfs_key, vfsmap[vfs_key], e) for (vfs_key, e) in entries]
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	774 _test_sync_point_walk_1_2(repo)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	775
50637 9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50636 diff changeset	776 max_linkrev = len(repo)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	777 file_count = totalfilesize = 0
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	778 with VolatileManager() as volatiles:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	779 # make sure we preserve volatile files
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	780 with util.nogc():
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	781 # record the expected size of every file
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	782 for k, vfs, e in entries:
52357 a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52356 diff changeset	783 e.preserve_volatiles(vfs, volatiles)
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	784 for f in e.files():
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	785 file_count += 1
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	786 totalfilesize += f.file_size(vfs)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	787
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	788 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	789 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	790 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	791 progress.update(0)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	792 with progress:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	793 # the first yield release the lock on the repository
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	794 yield file_count, totalfilesize
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	795 totalbytecount = 0
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	796
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	797 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	798 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	799 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	800 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	801 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	802 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	803 preserve_file_count=True,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	804 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	805 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	806 yield src
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	807 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	808 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	809 yield name
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	810 bytecount = 0
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	811 for chunk in stream:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	812 bytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	813 totalbytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	814 progress.update(totalbytecount)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	815 yield chunk
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	816 if bytecount != size:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	817 # Would most likely be caused by a race due to `hg
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	818 # strip` or a revlog split
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	819 msg = _(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	820 b'clone could only read %d bytes from %s, but '
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	821 b'expected %d bytes'
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	822 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	823 raise error.Abort(msg % (bytecount, name, size))
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	824
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	825
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	826 def _emit3(repo, entries) -> Iterator[bytes \| None]:
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	827 """actually emit the stream bundle (v3)"""
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	828 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	829 # we keep repo.vfs out of the map on purpose, ther are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	830 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	831 #
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	832 # this assert is duplicated (from _makemap) as authors might think this is
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	833 # fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	834 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	835 raise error.ProgrammingError(
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	836 'repo.vfs must not be added to vfsmap for security reasons'
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	837 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	838
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	839 # translate the vfs once
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	840 # we only turn this into a list for the `_test_sync`, this is not ideal
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	841 base_entries = list(entries)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	842 _test_sync_point_walk_1_2(repo)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	843 entries = []
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	844 with VolatileManager() as volatiles:
52012 e308439339e2 stream: rename TempCopyManager to VolatileManager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51859 diff changeset	845 # make sure we preserve volatile files
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	846 for vfs_key, e in base_entries:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	847 vfs = vfsmap[vfs_key]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	848 any_files = True
51549 49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	849 if e.maybe_volatile:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	850 any_files = False
52357 a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52356 diff changeset	851 e.preserve_volatiles(vfs, volatiles)
51549 49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	852 for f in e.files():
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	853 if f.is_volatile:
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	854 # record the expected size under lock
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51548 diff changeset	855 f.file_size(vfs)
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	856 any_files = True
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	857 if any_files:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	858 entries.append((vfs_key, vfsmap[vfs_key], e))
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	859
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	860 total_entry_count = len(entries)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	861
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	862 max_linkrev = len(repo)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	863 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	864 _(b'bundle'),
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	865 total=total_entry_count,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	866 unit=_(b'entry'),
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	867 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	868 progress.update(0)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	869 # the first yield release the lock on the repository
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	870 yield None
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	871 with progress:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	872 yield util.uvarintencode(total_entry_count)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	873
52356 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	874 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	875 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	876 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	877 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	878 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	879 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	880 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	881 yield util.uvarintencode(len(entry_streams))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	882 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	883 yield src
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	884 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	885 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	886 yield name
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	887 yield from stream
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52184 diff changeset	888 progress.increment()
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	889
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	890
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	891 def _test_sync_point_walk_1_2(repo):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	892 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	893
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	894 Triggered after gather entry, but before starting to process/preserve them
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	895 under lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	896
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	897 (on v1 is triggered before the actual walk start)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	898 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	899
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	900
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	901 def _test_sync_point_walk_3(repo):
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	902 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	903
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	904 Triggered right before releasing the lock, but after computing what need
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	905 needed to compute under lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	906 """
46986 faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	907
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	908
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	909 def _test_sync_point_walk_4(repo):
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	910 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	911
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	912 Triggered right after releasing the lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	913 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	914
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	915
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	916 # not really a StoreEntry, but close enough
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	917 class CacheEntry(store.SimpleStoreEntry):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	918 """Represent an entry for Cache files
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	919
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	920 It has special logic to preserve cache file early and accept optional
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	921 presence.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	922
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	923
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	924 (Yes... this is not really a StoreEntry, but close enough. We could have a
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	925 BaseEntry base class, bbut the store one would be identical)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	926 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	927
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	928 def __init__(self, entry_path) -> None:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	929 super().__init__(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	930 entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	931 # we will directly deal with that in `setup_cache_file`
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	932 is_volatile=True,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	933 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	934
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	935 def preserve_volatiles(self, vfs, volatiles) -> None:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	936 self._file_size = volatiles.try_keep(vfs.join(self._entry_path))
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	937 if self._file_size is None:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	938 self._files = []
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	939 else:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	940 assert self._is_volatile
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	941 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	942 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	943 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	944 file_size=self._file_size,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	945 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	946 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	947 ]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	948
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	949 def files(self) -> list[store.StoreFile]:
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	950 if self._files is None:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	951 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	952 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	953 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	954 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	955 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	956 ]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	957 return super().files()
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	958
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	959
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	960 class CacheFile(store.StoreFile):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	961 # inform the "copy/hardlink" version that this file might be missing
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	962 # without consequences.
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	963 optional: bool = True
46986 faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	964
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	965
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	966 def _entries_walk(repo, includes, excludes, includeobsmarkers: bool):
50512 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	967 """emit a seris of files information useful to clone a repo
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	968
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	969 return (vfs-key, entry) iterator
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	970
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	971 Where `entry` is StoreEntry. (used even for cache entries)
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	972 """
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	973 assert repo._currentlock(repo._lockref) is not None
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	974
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	975 matcher = None
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	976 if includes or excludes:
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	977 matcher = narrowspec.match(repo.root, includes, excludes)
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	978
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	979 phase = not repo.publishing()
51547 463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	980 # Python is getting crazy at all the small container we creates, disabling
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	981 # the gc while we do so helps performance a lot.
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	982 with util.nogc():
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	983 entries = _walkstreamfiles(
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	984 repo,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	985 matcher,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	986 phase=phase,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	987 obsolescence=includeobsmarkers,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	988 )
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	989 for entry in entries:
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	990 yield (_srcstore, entry)
50512 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	991
51547 463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	992 for name in cacheutil.cachetocopy(repo):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	993 if repo.cachevfs.exists(name):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51545 diff changeset	994 # not really a StoreEntry, but close enough
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	995 yield (_srccache, CacheEntry(entry_path=name))
50513 43ed1f12b00a stream-clone: yield cache entry in `_entries_walk` too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50512 diff changeset	996
50512 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50511 diff changeset	997
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	998 def generatev2(repo, includes, excludes, includeobsmarkers: bool):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	999 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1000
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1001 the data stream consists the following entries:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1002 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1003 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1004 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1005 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1006 5) N bytes containing the file data
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1007
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1008 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1009 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1010
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1011 with repo.lock():
47444 2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1012 repo.ui.debug(b'scanning\n')
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	1013
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	1014 entries = _entries_walk(
47444 2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1015 repo,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1016 includes=includes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1017 excludes=excludes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1018 includeobsmarkers=includeobsmarkers,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1019 )
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	1020 chunks = _emit2(repo, entries)
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	1021 first = next(chunks)
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	1022 file_count, total_file_size = first
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1023 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1024 _test_sync_point_walk_4(repo)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1025
50632 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50631 diff changeset	1026 return file_count, total_file_size, chunks
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1027
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1028
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1029 def generatev3(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1030 repo, includes, excludes, includeobsmarkers: bool
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1031 ) -> Iterator[bytes \| None]:
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1032 """Emit content for version 3 of a streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1033
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1034 the data stream consists the following:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1035 1) A varint E containing the number of entries (can be 0), then E entries follow
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1036 2) For each entry:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1037 2.1) The number of files in this entry (can be 0, but typically 1 or 2)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1038 2.2) For each file:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1039 2.2.1) A char representing the file destination (eg: store or cache)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1040 2.2.2) A varint N containing the length of the filename
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1041 2.2.3) A varint M containing the length of file data
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1042 2.2.4) N bytes containing the filename (the internal, store-agnostic form)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1043 2.2.5) M bytes containing the file data
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1044
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1045 Returns the data iterator.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1046
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1047 XXX This format is experimental and subject to change. Here is a
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1048 XXX non-exhaustive list of things this format could do or change:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1049
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1050 - making it easier to write files in parallel
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1051 - holding the lock for a shorter time
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1052 - improving progress information
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1053 - ways to adjust the number of expected entries/files ?
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1054 """
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1055
51548 6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1056 # Python is getting crazy at all the small container we creates while
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1057 # considering the files to preserve, disabling the gc while we do so helps
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1058 # performance a lot.
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51547 diff changeset	1059 with repo.lock(), util.nogc():
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1060 repo.ui.debug(b'scanning\n')
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1061
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1062 entries = _entries_walk(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1063 repo,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1064 includes=includes,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1065 excludes=excludes,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1066 includeobsmarkers=includeobsmarkers,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1067 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1068 chunks = _emit3(repo, list(entries))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1069 first = next(chunks)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1070 assert first is None
52358 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1071 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52357 diff changeset	1072 _test_sync_point_walk_4(repo)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1073
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1074 return chunks
50521 58adcabc295f stream-clone: introduce the notion of an experimental "v3" version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50520 diff changeset	1075
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50520 diff changeset	1076
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1077 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1078 def nested(*ctxs):
39757 97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1079 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1080 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1081 with this:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1082 if rest:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1083 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1084 yield
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39700 diff changeset	1085 else:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1086 yield
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1087
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1088
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1089 class V2Report:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1090 """a small class to track the data we saw within the stream"""
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1091
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1092 def __init__(self):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1093 self.byte_count = 0
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1094
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1095
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1096 def consumev2(repo, fp, filecount: int, filesize: int) -> None:
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1097 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1098
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1099 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1100 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1101 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1102 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1103 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1104 _(b'%d files to transfer, %s of data\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1105 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1106 )
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1107 progress = repo.ui.makeprogress(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1108 _(b'clone'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1109 total=filesize,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1110 unit=_(b'bytes'),
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1111 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1112 start = util.timer()
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1113 report = V2Report()
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1114
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1115 vfsmap = _makemap(repo)
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1116 # we keep repo.vfs out of the on purpose, ther are too many danger
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1117 # there (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1118 #
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1119 # this assert is duplicated (from _makemap) as author might think this
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1120 # is fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1121 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1122 raise error.ProgrammingError(
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	1123 'repo.vfs must not be added to vfsmap for security reasons'
47502 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1124 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1125
52913 58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	1126 cpu_profile = scmutil.get_resource_profile(repo.ui, b'cpu')
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1127 mem_profile = scmutil.get_resource_profile(repo.ui, b'memory')
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1128 threaded = repo.ui.configbool(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1129 b"worker", b"parallel-stream-bundle-processing"
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1130 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1131 num_writer = repo.ui.configint(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1132 b"worker",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1133 b"parallel-stream-bundle-processing.num-writer",
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1134 )
52913 58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	1135 if num_writer <= 0:
58baa86c7a02 stream-clone-v2: make the number of writer dependent of usage config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52912 diff changeset	1136 num_writer = DEFAULT_NUM_WRITER[cpu_profile]
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1137 memory_target = repo.ui.configbytes(
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1138 b"worker",
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1139 b"parallel-stream-bundle-processing.memory-target",
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1140 )
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1141 if memory_target < 0:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1142 memory_target = None
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1143 elif memory_target == 0:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1144 memory_target = DEFAULT_MEMORY_TARGET[mem_profile]
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1145 with repo.transaction(b'clone'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1146 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1147 with nested(*ctxs):
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1148 workers = []
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1149 info_queue = None
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1150 data_queue = None
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1151 mark_used = None
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1152 try:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1153 if not threaded:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1154 fc = _FileChunker
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1155 raw_data = fp
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1156 else:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1157 fc = _ThreadSafeFileChunker
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1158 data_queue = _DataQueue(memory_target=memory_target)
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1159 if memory_target is not None:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1160 mark_used = data_queue.mark_used
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1161 raw_data = util.chunkbuffer(data_queue)
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1162
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1163 w = threading.Thread(
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1164 target=data_queue.fill_from,
52917 9abf173a958b stream-clone-v2: use the pass through for threaded stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52914 diff changeset	1165 args=(fp,),
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1166 )
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1167 workers.append(w)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1168 w.start()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1169 files = _v2_parse_files(
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1170 repo,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1171 raw_data,
52929 5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52928 diff changeset	1172 vfsmap,
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1173 filecount,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1174 progress,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1175 report,
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1176 file_chunker=fc,
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1177 mark_used=mark_used,
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1178 )
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1179 if not threaded:
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1180 _write_files(files)
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1181 else:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1182 info_queue = _FileInfoQueue(files)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1183
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1184 for __ in range(num_writer):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1185 w = threading.Thread(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1186 target=_write_files,
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1187 args=(info_queue,),
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1188 )
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1189 workers.append(w)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1190 w.start()
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1191 info_queue.fill()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1192 except: # re-raises
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1193 if data_queue is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1194 data_queue.abort()
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1195 raise
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1196 finally:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1197 # shut down all the workers
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1198 if info_queue is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1199 # this is strictly speaking one too many worker for
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1200 # this queu, but closing too many is not a problem.
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1201 info_queue.close(len(workers))
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1202 for w in workers:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1203 w.join()
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1204
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1205 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1206 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1207 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1208
38373 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38360 diff changeset	1209 progress.complete()
52703 2e82bd50978c stream: acknowledge the end of the bundle part earlier Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52702 diff changeset	1210 # acknowledge the end of the bundle2 part, this help aligning
2e82bd50978c stream: acknowledge the end of the bundle part earlier Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52702 diff changeset	1211 # sequential and parallel behavior.
2e82bd50978c stream: acknowledge the end of the bundle part earlier Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52702 diff changeset	1212 remains = fp.read(1)
2e82bd50978c stream: acknowledge the end of the bundle part earlier Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52702 diff changeset	1213 assert not remains
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1214 _report_transferred(repo, start, filecount, report.byte_count)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1215
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1216
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1217 # iterator of chunk of bytes that constitute a file content.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1218 FileChunksT = Iterator[bytes]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1219 # Contains the information necessary to write stream file on disk
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1220 FileInfoT = Tuple[
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1221 bytes, # real fs path
22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1222 Optional[int], # permission to give to chmod
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1223 FileChunksT, # content
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1224 ]
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1225
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1226
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1227 class _Queue:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1228 """a reimplementation of queue.Queue which doesn't use thread.Condition"""
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1229
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1230 def __init__(self):
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1231 self._queue = collections.deque()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1232
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1233 # the "_lock" protect manipulation of the "_queue" deque
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1234 # the "_wait" is used to have the "get" thread waits for the
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1235 # "put" thread when the queue is empty.
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1236 #
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1237 # This is similar to the "threading.Condition", but without the absurd
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1238 # slowness of the stdlib implementation.
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1239 #
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1240 # the "_wait" is always released while holding the "_lock".
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1241 self._lock = threading.Lock()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1242 self._wait = threading.Lock()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1243
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1244 def put(self, item):
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1245 with self._lock:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1246 self._queue.append(item)
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1247 # if anyone is waiting on item, unblock it.
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1248 if self._wait.locked():
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1249 self._wait.release()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1250
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1251 def get(self):
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1252 with self._lock:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1253 while len(self._queue) == 0:
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1254 # "arm" the waiting lock
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1255 self._wait.acquire(blocking=False)
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1256 # release the lock to let other touch the queue
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1257 # (especially the put call we wait on)
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1258 self._lock.release()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1259 # wait for for a `put` call to release the lock
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1260 self._wait.acquire()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1261 # grab the lock to look at a possible available value
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1262 self._lock.acquire()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1263 # disarm the lock if necessary.
52923 f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52922 diff changeset	1264 #
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52922 diff changeset	1265 # If the queue only constains one item, keep the _wait lock
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52922 diff changeset	1266 # armed, as there is no need to wake another waiter anyway.
f1ac5117459b stream-clone-v2: avoid waking further thread if only 1 item is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52922 diff changeset	1267 if self._wait.locked() and len(self._queue) > 1:
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1268 self._wait.release()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1269 return self._queue.popleft()
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1270
0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1271
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1272 class _DataQueue:
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1273 """A queue passing data from the bundle stream to other thread
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1274
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1275 It has a "memory_target" optional parameter to avoid buffering too much
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1276 information. The implementation is not exact and the memory target might be
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1277 exceed for a time in some situation.
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1278 """
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1279
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1280 def __init__(self, memory_target=None):
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1281 self._q = _Queue()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1282 self._abort = False
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1283 self._memory_target = memory_target
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1284 if self._memory_target is not None and self._memory_target <= 0:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1285 raise error.ProgrammingError("memory target should be > 0")
52921 363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1286
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1287 # the "_lock" protect manipulation of the _current_used" variable
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1288 # the "_wait" is used to have the "reading" thread waits for the
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1289 # "using" thread when the buffer is full.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1290 #
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1291 # This is similar to the "threading.Condition", but without the absurd
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1292 # slowness of the stdlib implementation.
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1293 #
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1294 # the "_wait" is always released while holding the "_lock".
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1295 self._lock = threading.Lock()
363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1296 self._wait = threading.Lock()
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1297 # only the stream reader touch this, it is find to touch without the lock
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1298 self._current_read = 0
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1299 # do not touch this without the lock
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1300 self._current_used = 0
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1301
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1302 def _has_free_space(self):
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1303 """True if more data can be read without further exceeding memory target
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1304
52921 363914ba328d stream-clone-v2: no longer use the stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52920 diff changeset	1305 Must be called under the lock.
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1306 """
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1307 if self._memory_target is None:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1308 # Ideally we should not even get into the locking business in that
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1309 # case, but we keep the implementation simple for now.
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1310 return True
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1311 return (self._current_read - self._current_used) < self._memory_target
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1312
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1313 def mark_used(self, offset):
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1314 """Notify we have used the buffer up to "offset"
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1315
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1316 This is meant to be used from another thread than the one filler the queue.
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1317 """
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1318 if self._memory_target is not None:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1319 with self._lock:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1320 if offset > self._current_used:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1321 self._current_used = offset
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1322 # If the reader is waiting for room, unblock it.
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1323 if self._wait.locked() and self._has_free_space():
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1324 self._wait.release()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1325
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1326 def fill_from(self, data):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1327 """fill the data queue from a bundle2 part object
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1328
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1329 This is meant to be called by the data reading thread
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1330 """
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1331 q = self._q
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1332 try:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1333 for item in data:
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1334 self._current_read += len(item)
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1335 q.put(item)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1336 if self._abort:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1337 break
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1338 if self._memory_target is not None:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1339 with self._lock:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1340 while not self._has_free_space():
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1341 # make sure the _wait lock is locked
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1342 # this is done under lock, so there case be no race with the release logic
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1343 self._wait.acquire(blocking=False)
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1344 self._lock.release()
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1345 # acquiring the lock will block until some other thread release it.
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1346 self._wait.acquire()
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1347 # lets dive into the locked section again
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1348 self._lock.acquire()
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1349 # make sure we release the lock we just grabed if
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1350 # needed.
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1351 if self._wait.locked():
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1352 self._wait.release()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1353 finally:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1354 q.put(None)
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1355
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1356 def __iter__(self):
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1357 """Iterate over the bundle chunkgs
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1358
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1359 This is meant to be called by the data parsing thread."""
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1360 q = self._q
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1361 while (i := q.get()) is not None:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1362 yield i
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1363 if self._abort:
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1364 break
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1365
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1366 def abort(self):
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1367 """stop the data-reading thread and interrupt the comsuming iteration
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1368
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1369 This is meant to be called on errors.
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1370 """
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1371 self._abort = True
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1372 self._q.put(None)
52996 5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1373 if self._memory_target is not None:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1374 with self._lock:
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1375 # make sure we unstuck the reader thread.
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1376 if self._wait.locked():
5480647c2964 stream-clone: skip the memory_target logic when usage is unlimited Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52971 diff changeset	1377 self._wait.release()
52914 d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1378
d5ae681834e8 stream-clone-v2: also use a thread to read the streamed data Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52913 diff changeset	1379
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1380 class _FileInfoQueue:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1381 """A thread-safe queue to passer parsed file information to the writers"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1382
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1383 def __init__(self, info: Iterable[FileInfoT]):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1384 self._info = info
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1385 self._q = _Queue()
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1386
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1387 def fill(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1388 """iterate over the parsed information to file the queue
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1389
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1390 This is meant to be call from the thread parsing the stream information.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1391 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1392 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1393 for i in self._info:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1394 q.put(i)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1395
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1396 def close(self, number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1397 """signal all the workers that we no longer have any file info coming
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1398
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1399 Called from the thread parsing the stream information (and/or the main
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1400 thread if different).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1401 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1402 for __ in range(number_worker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1403 self._q.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1404
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1405 def __iter__(self):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1406 """iterate over the available file info
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1407
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1408 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1409 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1410 q = self._q
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1411 while (i := q.get()) is not None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1412 yield i
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1413
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1414
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1415 class _FileChunker:
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1416 """yield the chunk that constitute a file
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1417
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1418 This class exists as the counterpart of the threaded version and
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1419 would not be very useful on its own.
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1420 """
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1421
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1422 def __init__(
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1423 self,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1424 fp: bundle2mod.unbundlepart,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1425 data_len: int,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1426 progress: scmutil.progress,
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1427 report: V2Report,
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1428 mark_used: Optional[Callable[[int], None]] = None,
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1429 ):
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1430 self.report = report
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1431 self.progress = progress
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1432 self._chunks = util.filechunkiter(fp, limit=data_len)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1433
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1434 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1435 """Do nothing in non-threading context"""
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1436
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1437 def __iter__(self) -> FileChunksT:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1438 for chunk in self._chunks:
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1439 self.report.byte_count += len(chunk)
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1440 self.progress.increment(step=len(chunk))
307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1441 yield chunk
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1442
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1443
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1444 class _ThreadSafeFileChunker(_FileChunker):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1445 """yield the chunk that constitute a file
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1446
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1447 Make sure you call the "fill" function in the main thread to read the
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1448 right data at the right time.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1449 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1450
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1451 def __init__(
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1452 self,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1453 fp: bundle2mod.unbundlepart,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1454 data_len: int,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1455 progress: scmutil.progress,
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1456 report: V2Report,
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1457 mark_used: Optional[Callable[[int], None]] = None,
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1458 ):
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1459 super().__init__(fp, data_len, progress, report)
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1460 self._fp = fp
52922 0af8965b668a stream-clone-v2: use a Queue implementation without a stdlib Condition object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52921 diff changeset	1461 self._queue = _Queue()
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1462 self._mark_used = mark_used
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1463
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1464 def fill(self) -> None:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1465 """fill the file chunker queue with data read from the stream
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1466
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1467 This is meant to be called from the thread parsing information (and
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1468 consuming the stream data).
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1469 """
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1470 try:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1471 for chunk in super().__iter__():
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1472 offset = self._fp.tell()
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1473 self._queue.put((chunk, offset))
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1474 finally:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1475 self._queue.put(None)
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1476
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1477 def __iter__(self) -> FileChunksT:
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1478 """Iterate over all the file chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1479
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1480 This is meant to be called from the writer threads.
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1481 """
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1482 while (info := self._queue.get()) is not None:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1483 chunk, offset = info
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1484 if self._mark_used is not None:
aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1485 self._mark_used(offset)
52912 7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1486 yield chunk
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1487
7f848cfc4286 stream-clone-v2: use dedicated threads to write the data on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52911 diff changeset	1488
52925 119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1489 def _trivial_file(
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1490 chunk: bytes,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1491 mark_used: Optional[Callable[[int], None]],
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1492 offset: int,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1493 ) -> FileChunksT:
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1494 """used for single chunk file,"""
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1495 if mark_used is not None:
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1496 mark_used(offset)
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1497 yield chunk
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1498
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1499
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1500 def _v2_parse_files(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1501 repo,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1502 fp: bundle2mod.unbundlepart,
52929 5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52928 diff changeset	1503 vfs_map,
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1504 file_count: int,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1505 progress: scmutil.progress,
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1506 report: V2Report,
52911 307c4a0b91a0 stream-clone-v2: turn the file chunking function into a class Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52910 diff changeset	1507 file_chunker: Type[_FileChunker] = _FileChunker,
52920 aee193b1c784 stream-clone-v2: introduce a way to limit memory usage of the threaded version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52917 diff changeset	1508 mark_used: Optional[Callable[[int], None]] = None,
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1509 ) -> Iterator[FileInfoT]:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1510 """do the "stream-parsing" part of stream v2
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1511
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1512 The parsed information are yield result for consumption by the "writer"
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1513 """
52929 5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52928 diff changeset	1514 known_dirs = set() # set of directory that we know to exists
52909 3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1515 progress.update(0)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1516 for i in range(file_count):
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1517 src = util.readexactly(fp, 1)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1518 namelen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1519 datalen = util.uvarintdecodestream(fp)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1520
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1521 name = util.readexactly(fp, namelen)
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1522
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1523 if repo.ui.debugflag:
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1524 repo.ui.debug(
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1525 b'adding [%s] %s (%s)\n' % (src, name, util.bytecount(datalen))
3ee343dd3abf stream-clone-v2: extract the stream parsing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52703 diff changeset	1526 )
52929 5b8f6e198a6e stream-clone-v2: centralize preparation for streamed files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52928 diff changeset	1527 vfs = vfs_map[src]
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1528 path, mode = vfs.prepare_streamed_file(name, known_dirs)
52925 119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1529 if datalen <= util.DEFAULT_FILE_CHUNK:
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1530 c = fp.read(datalen)
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1531 offset = fp.tell()
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1532 report.byte_count += len(c)
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1533 progress.increment(step=len(c))
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1534 chunks = _trivial_file(c, mark_used, offset)
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1535 yield (path, mode, iter(chunks))
52925 119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1536 else:
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1537 chunks = file_chunker(
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1538 fp,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1539 datalen,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1540 progress,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1541 report,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1542 mark_used=mark_used,
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1543 )
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1544 yield (path, mode, iter(chunks))
52925 119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1545 # make sure we read all the chunk before moving to the next file
119cddd02b8c stream-clone-v2: simplify the handling of small file Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52923 diff changeset	1546 chunks.fill()
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1547
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1548
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1549 def _write_files(info: Iterable[FileInfoT]):
52910 70306aefa52b stream-clone-v2: extract the file writing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52909 diff changeset	1550 """write files from parsed data"""
52971 c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1551 io_flags = os.O_WRONLY \| os.O_CREAT
c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1552 if pycompat.iswindows:
c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1553 io_flags \|= os.O_BINARY
52930 22e264ac7f60 stream-clone-v2: bypass the vfs to write the file on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52929 diff changeset	1554 for path, mode, data in info:
52970 e4ff37b5317c stream-bundle: simple use of `mode` argument of `os.open` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52931 diff changeset	1555 if mode is None:
52971 c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1556 fd = os.open(path, io_flags)
52970 e4ff37b5317c stream-bundle: simple use of `mode` argument of `os.open` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52931 diff changeset	1557 else:
52971 c636f2835c95 stream-bundle: make sure to open the file in binary mode Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52970 diff changeset	1558 fd = os.open(path, io_flags, mode=mode)
52931 c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1559 try:
52910 70306aefa52b stream-clone-v2: extract the file writing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52909 diff changeset	1560 for chunk in data:
52931 c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1561 written = os.write(fd, chunk)
52928 7fc882f7fada stream-clone-v2: disable buffering when writing the files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52926 diff changeset	1562 # write missing pieces if the write was interrupted
7fc882f7fada stream-clone-v2: disable buffering when writing the files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52926 diff changeset	1563 while written < len(chunk):
52931 c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1564 written = os.write(fd, chunk[written:])
c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1565 finally:
c124308e3cd4 stream-clone-v2: directly use the os module for file operation Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52930 diff changeset	1566 os.close(fd)
52910 70306aefa52b stream-clone-v2: extract the file writing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52909 diff changeset	1567
70306aefa52b stream-clone-v2: extract the file writing code in a function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52909 diff changeset	1568
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1569 def consumev3(repo, fp) -> None:
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1570 """Apply the contents from a version 3 streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1571
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1572 Data is read from an object that only needs to provide a ``read(size)``
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1573 method.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1574 """
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1575 with repo.lock():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1576 start = util.timer()
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1577
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1578 entrycount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1579 repo.ui.status(_(b'%d entries to transfer\n') % (entrycount))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1580
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1581 progress = repo.ui.makeprogress(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1582 _(b'clone'),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1583 total=entrycount,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1584 unit=_(b'entries'),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1585 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1586 progress.update(0)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1587 bytes_transferred = 0
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1588
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1589 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1590 # we keep repo.vfs out of the on purpose, there are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1591 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1592 #
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1593 # this assert is duplicated (from _makemap) as authors might think this
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1594 # is fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1595 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1596 raise error.ProgrammingError(
52696 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52695 diff changeset	1597 'repo.vfs must not be added to vfsmap for security reasons'
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1598 )
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	1599 total_file_count = 0
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1600 with repo.transaction(b'clone'):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1601 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1602 with nested(*ctxs):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1603 for i in range(entrycount):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1604 filecount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1605 if filecount == 0:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1606 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1607 repo.ui.debug(b'entry with no files [%d]\n' % (i))
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	1608 total_file_count += filecount
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1609 for i in range(filecount):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1610 src = util.readexactly(fp, 1)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1611 vfs = vfsmap[src]
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1612 namelen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1613 datalen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1614
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1615 name = util.readexactly(fp, namelen)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1616
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1617 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1618 msg = b'adding [%s] %s (%s)\n'
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1619 msg %= (src, name, util.bytecount(datalen))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1620 repo.ui.debug(msg)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1621 bytes_transferred += datalen
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1622
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1623 with vfs(name, b'w') as ofp:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1624 for chunk in util.filechunkiter(fp, limit=datalen):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1625 ofp.write(chunk)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1626 progress.increment(step=1)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1627
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1628 # force @filecache properties to be reloaded from
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1629 # streamclone-ed file at next access
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1630 repo.invalidate(clearfilecache=True)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1631
52701 6feb3b3029b5 stream: consistently close progress before reporting time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52700 diff changeset	1632 progress.complete()
52702 ca17d31624ac stream: report number of file written then information is available Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52701 diff changeset	1633 _report_transferred(repo, start, total_file_count, bytes_transferred)
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1634
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1635
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1636 def applybundlev2(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1637 repo, fp, filecount: int, filesize: int, requirements: Iterable[bytes]
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1638 ) -> None:
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	1639 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	1640
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1641 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1642 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1643 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	1644 _(b'unable to apply stream clone: unsupported format: %s')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1645 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1646 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35491 diff changeset	1647
52926 34fa51c25112 stream-clone-v2: disable the garbage collector during consumption Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52925 diff changeset	1648 with util.nogc():
34fa51c25112 stream-clone-v2: disable the garbage collector during consumption Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52925 diff changeset	1649 consumev2(repo, fp, filecount, filesize)
35804 2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	1650
48596 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	1651 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	1652 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47878 diff changeset	1653 requirements,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1654 )
39700 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38783 diff changeset	1655 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1656 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42813 diff changeset	1657 )
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	1658 scmutil.writereporequirements(repo)
48693 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48681 diff changeset	1659 nodemap.post_stream_cleanup(repo)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1660
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1661
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1662 def applybundlev3(repo, fp, requirements: Iterable[bytes]) -> None:
50703 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1663 from . import localrepo
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1664
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1665 missingreqs = [r for r in requirements if r not in repo.supported]
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1666 if missingreqs:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1667 msg = _(b'unable to apply stream clone: unsupported format: %s')
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1668 msg %= b', '.join(sorted(missingreqs))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1669 raise error.Abort(msg)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1670
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1671 consumev3(repo, fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1672
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1673 repo.requirements = new_stream_clone_requirements(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1674 repo.requirements,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1675 requirements,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1676 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1677 repo.svfs.options = localrepo.resolvestorevfsoptions(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1678 repo.ui, repo.requirements, repo.features
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1679 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1680 scmutil.writereporequirements(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1681 nodemap.post_stream_cleanup(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1682
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50663 diff changeset	1683
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1684 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress) -> bool:
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1685 hardlink = [True]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1686
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1687 def copy_used():
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1688 hardlink[0] = False
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1689 progress.topic = _(b'copying')
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1690
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1691 for k, path, optional in entries:
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1692 src_vfs = src_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1693 dst_vfs = dst_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1694 src_path = src_vfs.join(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1695 dst_path = dst_vfs.join(path)
47871 132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1696 # We cannot use dirname and makedirs of dst_vfs here because the store
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1697 # encoding confuses them. See issue 6581 for details.
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1698 dirname = os.path.dirname(dst_path)
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1699 if not os.path.exists(dirname):
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47788 diff changeset	1700 util.makedirs(dirname)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1701 dst_vfs.register_file(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1702 # XXX we could use the #nb_bytes argument.
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1703 try:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1704 util.copyfile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1705 src_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1706 dst_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1707 hardlink=hardlink[0],
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1708 no_hardlink_cb=copy_used,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1709 check_fs_hardlink=False,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1710 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1711 except FileNotFoundError:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1712 if not optional:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1713 raise
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1714 progress.increment()
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1715 return hardlink[0]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1716
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1717
52695 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52691 diff changeset	1718 def local_copy(src_repo, dest_repo) -> None:
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1719 """copy all content from one local repository to another
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1720
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1721 This is useful for local clone"""
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1722 src_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1723 r
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1724 for r in src_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1725 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1726 }
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1727 dest_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1728 r
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1729 for r in dest_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1730 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1731 }
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1732 assert src_store_requirements == dest_store_requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1733
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1734 with dest_repo.lock():
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1735 with src_repo.lock():
47448 d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1736 # bookmark is not integrated to the streaming as it might use the
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1737 # `repo.vfs` and they are too many sentitive data accessible
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1738 # through `repo.vfs` to expose it to streaming clone.
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1739 src_book_vfs = bookmarks.bookmarksvfs(src_repo)
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1740 srcbookmarks = src_book_vfs.join(b'bookmarks')
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1741 bm_count = 0
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1742 if os.path.exists(srcbookmarks):
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1743 bm_count = 1
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1744
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1745 entries = _entries_walk(
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1746 src_repo,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1747 includes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1748 excludes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1749 includeobsmarkers=True,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1750 )
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1751 entries = list(entries)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1752 src_vfs_map = _makemap(src_repo)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1753 dest_vfs_map = _makemap(dest_repo)
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1754 total_files = sum(len(e[1].files()) for e in entries) + bm_count
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1755 progress = src_repo.ui.makeprogress(
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1756 topic=_(b'linking'),
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1757 total=total_files,
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1758 unit=_(b'files'),
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1759 )
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1760 # copy files
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1761 #
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1762 # We could copy the full file while the source repository is locked
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1763 # and the other one without the lock. However, in the linking case,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1764 # this would also requires checks that nobody is appending any data
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1765 # to the files while we do the clone, so this is not done yet. We
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1766 # could do this blindly when copying files.
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1767 files = [
52359 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52358 diff changeset	1768 (vfs_key, f.unencoded_path, f.optional)
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1769 for vfs_key, e in entries
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1770 for f in e.files()
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1771 ]
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1772 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1773
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1774 # copy bookmarks over
47448 d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1775 if bm_count:
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1776 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47447 diff changeset	1777 dstbookmarks = dst_book_vfs.join(b'bookmarks')
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1778 util.copyfile(srcbookmarks, dstbookmarks)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1779 progress.complete()
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1780 if hardlink:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1781 msg = b'linked %d files\n'
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1782 else:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1783 msg = b'copied %d files\n'
50630 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50627 diff changeset	1784 src_repo.ui.debug(msg % total_files)
47447 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1785
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1786 with dest_repo.transaction(b"localclone") as tr:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1787 dest_repo.store.write(tr)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1788
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47444 diff changeset	1789 # clean up transaction file as they do not make sense
50288 d89eecf9605e undo-files: no longer pass the `repo` to `cleanup_undo_files` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50286 diff changeset	1790 transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)

Mercurial > public > mercurial-scm > hg

annotate mercurial/streamclone.py @ 53042:cdd7bf612c7b stable tip