mercurial-scm/hg-stable: mercurial/streamclone.py annotate

annotate mercurial/streamclone.py @ 52719:6feb3b3029b5

stream: consistently close progress before reporting time Seems more logical. We also take this as an opportunity to explicitly track bytes count in "v2" as relying on progress internal start is error prone.

author	Pierre-Yves David <pierre-yves.david@octobus.net>
date	Thu, 28 Nov 2024 11:05:07 +0100
parents	061bfd699a56
children	ca17d31624ac

rev	line source
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	1 # streamclone.py - producing and consuming streaming repository data
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	2 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	3 # Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	4 #
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	7
51901 f4733654f144 typing: add `from __future__ import annotations` to most files Matt Harbison <matt_harbison@yahoo.com> parents: 51762 diff changeset	8 from __future__ import annotations
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	9
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	10 import contextlib
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	11 import errno
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	12 import os
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	13 import struct
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	14
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	15 from typing import Iterable, Iterator, Optional, Set
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	16
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	17 from .i18n import _
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	18 from .interfaces import repository
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	19 from . import (
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	20 bookmarks,
50557 a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50556 diff changeset	21 bundle2 as bundle2mod,
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	22 cacheutil,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	23 error,
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	24 narrowspec,
32764 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	25 phases,
38197 aac4be30e250 py3: wrap tempfile.mkstemp() to use bytes path Yuya Nishihara <yuya@tcha.org> parents: 37638 diff changeset	26 pycompat,
46665 ee91966aec0f requirements: add constant for revlog v1 requirement Rapha?l Gom?s <rgomes@octobus.net> parents: 45106 diff changeset	27 requirements as requirementsmod,
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	28 scmutil,
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	29 store,
50286 3d0b5760851c undo-files: move the undo cleanup code in the transaction module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50279 diff changeset	30 transaction,
26442 ef8d27f53204 streamclone: move stream_in() from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26441 diff changeset	31 util,
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	32 )
48707 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48695 diff changeset	33 from .revlogutils import (
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48695 diff changeset	34 nodemap,
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48695 diff changeset	35 )
26441 56527b886d1d streamclone: move applystreamclone() from localrepo.py Gregory Szorc <gregory.szorc@gmail.com> parents: diff changeset	36
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	37
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	38 def new_stream_clone_requirements(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	39 default_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	40 streamed_requirements: Iterable[bytes],
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	41 ) -> Set[bytes]:
48618 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	42 """determine the final set of requirement for a new stream clone
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	43
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	44 this method combine the "default" requirements that a new repository would
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	45 use with the constaint we get from the stream clone content. We keep local
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	46 configuration choice when possible.
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	47 """
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	48 requirements = set(default_requirements)
48623 baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48622 diff changeset	49 requirements -= requirementsmod.STREAM_FIXED_REQUIREMENTS
48618 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	50 requirements.update(streamed_requirements)
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	51 return requirements
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	52
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	53
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	54 def streamed_requirements(repo) -> Set[bytes]:
48619 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48618 diff changeset	55 """the set of requirement the new clone will have to support
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48618 diff changeset	56
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48618 diff changeset	57 This is used for advertising the stream options and to generate the actual
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48618 diff changeset	58 stream content."""
48623 baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48622 diff changeset	59 requiredformats = (
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48622 diff changeset	60 repo.requirements & requirementsmod.STREAM_FIXED_REQUIREMENTS
baddab229b86 stream-clone: add a explicit set list requirements relevant to stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48622 diff changeset	61 )
48619 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48618 diff changeset	62 return requiredformats
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48618 diff changeset	63
8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48618 diff changeset	64
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	65 def canperformstreamclone(pullop, bundle2: bool = False):
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	66 """Whether it is possible to perform a streaming clone as part of pull.
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	67
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	68 ``bundle2`` will cause the function to consider stream clone through
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	69 bundle2 and only through bundle2.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	70
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	71 Returns a tuple of (supported, requirements). ``supported`` is True if
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	72 streaming clone is supported and False otherwise. ``requirements`` is
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	73 a set of repo requirements from the remote, or ``None`` if stream clone
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	74 isn't supported.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	75 """
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	76 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	77 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	78
50556 58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	79 # should we consider streaming clone at all ?
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	80 streamrequested = pullop.streamclonerequested
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	81 # If we don't have a preference, let the server decide for us. This
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	82 # likely only comes into play in LANs.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	83 if streamrequested is None:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	84 # The server can advertise whether to prefer streaming clone.
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	85 streamrequested = remote.capable(b'stream-preferred')
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	86 if not streamrequested:
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	87 return False, None
58e4842fbfc1 stream-clone: bail-out earlier if stream clone is not requested Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50555 diff changeset	88
50554 0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50550 diff changeset	89 # Streaming clone only works on an empty destination repository
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50550 diff changeset	90 if len(repo):
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50550 diff changeset	91 return False, None
0558866957fa stream-clone: bail-out earlier if destination repo is not empty Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50550 diff changeset	92
50555 f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50554 diff changeset	93 # Streaming clone only works if all data is being requested.
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50554 diff changeset	94 if pullop.heads:
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50554 diff changeset	95 return False, None
f697af015683 stream-clone: bail-out earlier if pull is partial Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50554 diff changeset	96
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	97 bundle2supported = False
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	98 if pullop.canusebundle2:
50557 a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50556 diff changeset	99 local_caps = bundle2mod.getrepocaps(repo, role=b'client')
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50556 diff changeset	100 local_supported = set(local_caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50556 diff changeset	101 remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
a6543983b8f4 stream-clone: check is a compatible protocol can be found Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50556 diff changeset	102 bundle2supported = bool(local_supported & remote_supported)
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	103 # else
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	104 # Server doesn't support bundle2 stream clone or doesn't support
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	105 # the versions we support. Fall back and possibly allow legacy.
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	106
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	107 # Ensures legacy code path uses available bundle2.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	108 if bundle2supported and not bundle2:
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	109 return False, None
ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	110 # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
35757 bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	111 elif bundle2 and not bundle2supported:
bbf7abd09ff0 streamclone: rework canperformstreamclone Boris Feld <boris.feld@octobus.net> parents: 35756 diff changeset	112 return False, None
26467 ff2c89239d49 streamclone: teach canperformstreamclone to be bundle2 aware Gregory Szorc <gregory.szorc@gmail.com> parents: 26466 diff changeset	113
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	114 # In order for stream clone to work, the client has to support all the
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	115 # requirements advertised by the server.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	116 #
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	117 # The server advertises its requirements via the "stream" and "streamreqs"
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	118 # capability. "stream" (a value-less capability) is advertised if and only
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	119 # if the only requirement is "revlogv1." Else, the "streamreqs" capability
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	120 # is advertised and contains a comma-delimited list of requirements.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	121 requirements = set()
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	122 if remote.capable(b'stream'):
46665 ee91966aec0f requirements: add constant for revlog v1 requirement Rapha?l Gom?s <rgomes@octobus.net> parents: 45106 diff changeset	123 requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	124 else:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	125 streamreqs = remote.capable(b'streamreqs')
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	126 # This is weird and shouldn't happen with modern servers.
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	127 if not streamreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	128 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	129 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	130 b'warning: stream clone requested but server has them '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	131 b'disabled\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	132 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	133 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	134 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	135
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	136 streamreqs = set(streamreqs.split(b','))
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	137 # Server requires something we don't support. Bail.
48359 6d2ddea0721a stream-clone: filter possible missing requirements using all supported one Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	138 missingreqs = streamreqs - repo.supported
32299 076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30995 diff changeset	139 if missingreqs:
076f1ff43f0f clone: warn when streaming was requested but couldn't be performed Siddharth Agarwal <sid0@fb.com> parents: 30995 diff changeset	140 pullop.repo.ui.warn(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	141 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	142 b'warning: stream clone requested but client is missing '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	143 b'requirements: %s\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	144 )
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	145 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	146 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	147 pullop.repo.ui.warn(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	148 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	149 b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	150 b'for more information)\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	151 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	152 )
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	153 return False, None
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	154 requirements = streamreqs
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	155
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	156 return True, requirements
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	157
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	158
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	159 def maybeperformlegacystreamclone(pullop) -> None:
26462 3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	160 """Possibly perform a legacy stream clone operation.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	161
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	162 Legacy stream clones are performed as part of pull but before all other
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	163 operations.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	164
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	165 A legacy stream clone will not be performed if a bundle2 stream clone is
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	166 supported.
3b0ec09192ae streamclone: rename and document maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26461 diff changeset	167 """
39716 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	168 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	169
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	170 supported, requirements = canperformstreamclone(pullop)
26458 362793295640 streamclone: refactor maybeperformstreamclone to take a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26447 diff changeset	171
26446 3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	172 if not supported:
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	173 return
3ea10bb761ce streamclone: refactor code for deciding to stream clone Gregory Szorc <gregory.szorc@gmail.com> parents: 26445 diff changeset	174
26466 3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	175 repo = pullop.repo
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	176 remote = pullop.remote
3515db5aae05 streamclone: refactor canperformstreamclone to accept a pullop Gregory Szorc <gregory.szorc@gmail.com> parents: 26462 diff changeset	177
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	178 # Save remote branchmap. We will use it later to speed up branchcache
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	179 # creation.
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	180 rbranchmap = None
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	181 if remote.capable(b'branchmap'):
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	182 with remote.commandexecutor() as e:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	183 rbranchmap = e.callcommand(b'branchmap', {}).result()
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	184
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	185 repo.ui.status(_(b'streaming all changes\n'))
26470 4b5647d9ee13 streamclone: move "streaming all changes" message location Gregory Szorc <gregory.szorc@gmail.com> parents: 26469 diff changeset	186
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	187 with remote.commandexecutor() as e:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	188 fp = e.callcommand(b'stream_out', {}).result()
37638 65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	189
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	190 # TODO strictly speaking, this code should all be inside the context
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	191 # manager because the context manager is supposed to ensure all wire state
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	192 # is flushed when exiting. But the legacy peers don't do this, so it
65b86ee69383 streamclone: use command executor for wire protocol commands Gregory Szorc <gregory.szorc@gmail.com> parents: 35804 diff changeset	193 # doesn't matter.
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	194 l = fp.readline()
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	195 try:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	196 resp = int(l)
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	197 except ValueError:
3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	198 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	199 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	200 )
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	201 if resp == 1:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	202 raise error.Abort(_(b'operation forbidden by server'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	203 elif resp == 2:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	204 raise error.Abort(_(b'locking the remote repository failed'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	205 elif resp != 0:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	206 raise error.Abort(_(b'the server sent an unknown error code'))
26459 3b28ffde133a streamclone: move streamin() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26458 diff changeset	207
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	208 l = fp.readline()
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	209 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	210 filecount, bytecount = map(int, l.split(b' ', 1))
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	211 except (ValueError, TypeError):
19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	212 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	213 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	214 )
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	215
27850 49cfddbf54ba with: use context manager in maybeperformlegacystreamclone Bryan O'Sullivan <bryano@fb.com> parents: 27845 diff changeset	216 with repo.lock():
26468 19bbd53af46d streamclone: move payload header line consumption Gregory Szorc <gregory.szorc@gmail.com> parents: 26467 diff changeset	217 consumev1(repo, fp, filecount, bytecount)
48618 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	218 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	219 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	220 requirements,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	221 )
39716 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	222 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	223 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	224 )
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	225 scmutil.writereporequirements(repo)
48707 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48695 diff changeset	226 nodemap.post_stream_cleanup(repo)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	227
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	228 if rbranchmap:
41626 328ca3b9e545 branchmap: encapsulate cache updating in the map itself Martijn Pieters <mj@octobus.net> parents: 40544 diff changeset	229 repo._branchcaches.replace(repo, rbranchmap)
26461 09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	230
09cc3c2e9ece streamclone: move applyremotedata() into maybeperformstreamclone() Gregory Szorc <gregory.szorc@gmail.com> parents: 26460 diff changeset	231 repo.invalidate()
26445 f134fb33c906 streamclone: move streaming clone logic from localrepo Gregory Szorc <gregory.szorc@gmail.com> parents: 26444 diff changeset	232
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	233
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	234 def allowservergeneration(repo) -> bool:
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	235 """Whether streaming clones are allowed from the server."""
40029 51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39855 diff changeset	236 if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39855 diff changeset	237 return False
51f10e6d66c7 streamclone: don't support stream clone unless repo feature present Gregory Szorc <gregory.szorc@gmail.com> parents: 39855 diff changeset	238
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	239 if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
32764 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	240 return False
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	241
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	242 # The way stream clone works makes it impossible to hide secret changesets.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	243 # So don't allow this by default.
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	244 secret = phases.hassecret(repo)
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	245 if secret:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	246 return repo.ui.configbool(b'server', b'uncompressedallowsecret')
32764 33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	247
33b7283a3828 streamclone: consider secret changesets (BC) (issue5589) Gregory Szorc <gregory.szorc@gmail.com> parents: 32299 diff changeset	248 return True
26444 623743010133 streamclone: move _allowstream() from wireproto Gregory Szorc <gregory.szorc@gmail.com> parents: 26443 diff changeset	249
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	250
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	251 # This is it's own function so extensions can override it.
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	252 def _walkstreamfiles(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	253 repo, matcher=None, phase: bool = False, obsolescence: bool = False
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	254 ):
50548 0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50546 diff changeset	255 return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	256
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	257
52718 061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	258 def _report_transferred(repo, start_time: float, byte_count: int):
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	259 """common utility to report time it took to download the stream bundle"""
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	260 elapsed = util.timer() - start_time
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	261 if elapsed <= 0:
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	262 elapsed = 0.001
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	263 m = _(b'transferred %s in %.1f seconds (%s/sec)\n')
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	264 m %= (
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	265 util.bytecount(byte_count),
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	266 elapsed,
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	267 util.bytecount(byte_count / elapsed),
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	268 )
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	269 repo.ui.status(m)
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	270
061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	271
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	272 def generatev1(repo) -> tuple[int, int, Iterator[bytes]]:
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	273 """Emit content for version 1 of a streaming clone.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	274
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	275 This returns a 3-tuple of (file count, byte size, data iterator).
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	276
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	277 The data iterator consists of N entries for each file being transferred.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	278 Each file entry starts as a line with the file name and integer size
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	279 delimited by a null byte.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	280
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	281 The raw file data follows. Following the raw file data is the next file
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	282 entry, or EOF.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	283
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	284 When used on the wire protocol, an additional line indicating protocol
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	285 success will be prepended to the stream. This function is not responsible
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	286 for adding it.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	287
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	288 This function will obtain a repository lock to ensure a consistent view of
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	289 the store is captured. It therefore may raise LockError.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	290 """
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	291 entries = []
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	292 total_bytes = 0
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	293 # Get consistent snapshot of repo, lock during scan.
27845 7417e1c10253 with: use context manager in streamclone generatev1 Bryan O'Sullivan <bryano@fb.com> parents: 27794 diff changeset	294 with repo.lock():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	295 repo.ui.debug(b'scanning\n')
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	296 _test_sync_point_walk_1_2(repo)
50505 521fec115dad store: use a StoreEntry object instead of tuple for store files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50288 diff changeset	297 for entry in _walkstreamfiles(repo):
50506 9fdc28e21b68 store: introduce a EntryFile object to actually access file info Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50505 diff changeset	298 for f in entry.files():
50511 4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50506 diff changeset	299 file_size = f.file_size(repo.store.vfs)
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50506 diff changeset	300 if file_size:
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50506 diff changeset	301 entries.append((f.unencoded_path, file_size))
4cbdfab6f812 store: lazily get file size on demand for the fncache case Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50506 diff changeset	302 total_bytes += file_size
52390 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52389 diff changeset	303 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52389 diff changeset	304 _test_sync_point_walk_4(repo)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	305
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	306 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	307 b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	308 )
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	309
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	310 svfs = repo.svfs
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	311 debugflag = repo.ui.debugflag
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	312
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	313 def emitrevlogdata() -> Iterator[bytes]:
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	314 for name, size in entries:
761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	315 if debugflag:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	316 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	317 # partially encode name over the wire for backwards compat
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	318 yield b'%s\0%d\n' % (store.encodedir(name), size)
33411 50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	319 # auditing at this stage is both pointless (paths are already
50b49bb0fff3 streamclone: comment why path auditing is disabled in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33410 diff changeset	320 # trusted by the local repo) and expensive
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	321 with svfs(name, b'rb', auditpath=False) as fp:
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	322 if size <= 65536:
33258 761ccfeff8b1 streamclone: stop using 'vfs.mustaudit = False' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33257 diff changeset	323 yield fp.read(size)
33410 c784308305c6 streamclone: close large revlog files explicitly in generatev1() Yuya Nishihara <yuya@tcha.org> parents: 33258 diff changeset	324 else:
52669 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52665 diff changeset	325 yield from util.filechunkiter(fp, limit=size)
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	326
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	327 return len(entries), total_bytes, emitrevlogdata()
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	328
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	329
52709 279e217d6041 typing: lock in the new type annotations detected with the pyupgrade changes Matt Harbison <matt_harbison@yahoo.com> parents: 52669 diff changeset	330 def generatev1wireproto(repo) -> Iterator[bytes]:
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	331 """Emit content for version 1 of streaming clone suitable for the wire.
fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	332
35495 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	333 This is the data output from ``generatev1()`` with 2 header lines. The
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	334 first line indicates overall success. The 2nd contains the file count and
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	335 byte size of payload.
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	336
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	337 The success line contains "0" for success, "1" for stream generation not
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	338 allowed, and "2" for error locking the repository (possibly indicating
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	339 a permissions error for the server process).
26469 fb743268510e streamclone: move payload header generation into own function Gregory Szorc <gregory.szorc@gmail.com> parents: 26468 diff changeset	340 """
35495 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	341 if not allowservergeneration(repo):
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	342 yield b'1\n'
35495 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	343 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	344
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	345 try:
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	346 filecount, bytecount, it = generatev1(repo)
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	347 except error.LockError:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	348 yield b'2\n'
35495 ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	349 return
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	350
ded3a63f305b streamclone: move wire protocol status code from wireproto command Gregory Szorc <gregory.szorc@gmail.com> parents: 33499 diff changeset	351 # Indicates successful response.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	352 yield b'0\n'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	353 yield b'%d %d\n' % (filecount, bytecount)
52669 e627cc25b6f3 pyupgrade: rewrite `yield` statements in a loop to `yield from` Matt Harbison <matt_harbison@yahoo.com> parents: 52665 diff changeset	354 yield from it
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	355
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	356
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	357 def generatebundlev1(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	358 repo, compression: bytes = b'UN'
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	359 ) -> tuple[Set[bytes], Iterator[bytes]]:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	360 """Emit content for version 1 of a stream clone bundle.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	361
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	362 The first 4 bytes of the output ("HGS1") denote this as stream clone
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	363 bundle version 1.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	364
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	365 The next 2 bytes indicate the compression type. Only "UN" is currently
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	366 supported.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	367
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	368 The next 16 bytes are two 64-bit big endian unsigned integers indicating
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	369 file count and byte count, respectively.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	370
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	371 The next 2 bytes is a 16-bit big endian unsigned short declaring the length
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	372 of the requirements string, including a trailing \0. The following N bytes
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	373 are the requirements string, which is ASCII containing a comma-delimited
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	374 list of repo requirements that are needed to support the data.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	375
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	376 The remaining content is the output of ``generatev1()`` (which may be
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	377 compressed in the future).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	378
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	379 Returns a tuple of (requirements, data generator).
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	380 """
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	381 if compression != b'UN':
52714 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52713 diff changeset	382 raise ValueError('we do not support the compression argument yet')
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	383
48619 8475a1364909 stream-clone: factor computation of requirement of a stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48618 diff changeset	384 requirements = streamed_requirements(repo)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	385 requires = b','.join(sorted(requirements))
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	386
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	387 def gen() -> Iterator[bytes]:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	388 yield b'HGS1'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	389 yield compression
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	390
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	391 filecount, bytecount, it = generatev1(repo)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	392 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	393 _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	394 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	395
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	396 yield struct.pack(b'>QQ', filecount, bytecount)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	397 yield struct.pack(b'>H', len(requires) + 1)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	398 yield requires + b'\0'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	399
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	400 # This is where we'll add compression in the future.
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	401 assert compression == b'UN'
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	402
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	403 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	404 _(b'bundle'), total=bytecount, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	405 )
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	406 progress.update(0)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	407
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	408 for chunk in it:
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	409 progress.increment(step=len(chunk))
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	410 yield chunk
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	411
38379 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38366 diff changeset	412 progress.complete()
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	413
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	414 return requirements, gen()
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	415
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	416
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	417 def consumev1(repo, fp, filecount: int, bytecount: int) -> None:
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	418 """Apply the contents from version 1 of a streaming clone file handle.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	419
30342 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29923 diff changeset	420 This takes the output from "stream_out" and applies it to the specified
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	421 repository.
d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	422
30342 318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29923 diff changeset	423 Like "stream_out," the status line added by the wire protocol is not
318a24b52eeb spelling: fixes of non-dictionary words Mads Kiilerich <madski@unity3d.com> parents: 29923 diff changeset	424 handled by this function.
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	425 """
27859 f55a5ace8e69 with: use context manager in streamclone consumev1 Bryan O'Sullivan <bryano@fb.com> parents: 27850 diff changeset	426 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	427 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	428 _(b'%d files to transfer, %s of data\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	429 % (filecount, util.bytecount(bytecount))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	430 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	431 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	432 _(b'clone'), total=bytecount, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	433 )
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	434 progress.update(0)
30995 22fbca1d11ed mercurial: switch to util.timer for all interval timings Simon Farnsworth <simonfar@fb.com> parents: 30342 diff changeset	435 start = util.timer()
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	436
29923 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	437 # TODO: get rid of (potential) inconsistency
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	438 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	439 # If transaction is started and any @filecache property is
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	440 # changed at this point, it causes inconsistency between
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	441 # in-memory cached property and streamclone-ed file on the
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	442 # disk. Nested transaction prevents transaction scope "clone"
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	443 # below from writing in-memory changes out at the end of it,
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	444 # even though in-memory changes are discarded at the end of it
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	445 # regardless of transaction nesting.
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	446 #
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	447 # But transaction nesting can't be simply prohibited, because
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	448 # nesting occurs also in ordinary case (e.g. enabling
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	449 # clonebundles).
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	450
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	451 with repo.transaction(b'clone'):
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	452 with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
49292 d44e3c45f0e4 py3: replace `pycompat.xrange` by `range` Manuel Jacob <me@manueljacob.de> parents: 49222 diff changeset	453 for i in range(filecount):
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	454 # XXX doesn't support '\n' or '\r' in filenames
50951 d718eddf01d9 safehasattr: drop usage in favor of hasattr Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50706 diff changeset	455 if hasattr(fp, 'readline'):
50675 60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50670 diff changeset	456 l = fp.readline()
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50670 diff changeset	457 else:
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50670 diff changeset	458 # inline clonebundles use a chunkbuffer, so no readline
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50670 diff changeset	459 # --> this should be small anyway, the first line
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50670 diff changeset	460 # only contains the size of the bundle
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50670 diff changeset	461 l_buf = []
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50670 diff changeset	462 while not (l_buf and l_buf[-1] == b'\n'):
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50670 diff changeset	463 l_buf.append(fp.read(1))
60f9602b413e clonebundles: add support for inline (streaming) clonebundles Mathias De Mare <mathias.de_mare@nokia.com> parents: 50670 diff changeset	464 l = b''.join(l_buf)
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	465 try:
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	466 name, size = l.split(b'\0', 1)
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	467 size = int(size)
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	468 except (ValueError, TypeError):
1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	469 raise error.ResponseError(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	470 _(b'unexpected response from remote server:'), l
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	471 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	472 if repo.ui.debugflag:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	473 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	474 b'adding %s (%s)\n' % (name, util.bytecount(size))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	475 )
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	476 # for backwards compat, name was partially encoded
27897 2fdbf22a1b63 streamclone: use backgroundfilecloser (issue4889) Gregory Szorc <gregory.szorc@gmail.com> parents: 27896 diff changeset	477 path = store.decodedir(name)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	478 with repo.svfs(path, b'w', backgroundclose=True) as ofp:
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	479 for chunk in util.filechunkiter(fp, limit=size):
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	480 progress.increment(step=len(chunk))
27896 1d29893240cc streamclone: indent code Gregory Szorc <gregory.szorc@gmail.com> parents: 27882 diff changeset	481 ofp.write(chunk)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	482
29923 519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	483 # force @filecache properties to be reloaded from
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	484 # streamclone-ed file at next access
519a02267f90 streamclone: clear caches after writing changes into files for visibility FUJIWARA Katsunori <foozy@lares.dti.ne.jp> parents: 29921 diff changeset	485 repo.invalidate(clearfilecache=True)
26443 d947086d8973 streamclone: move code out of exchange.py Gregory Szorc <gregory.szorc@gmail.com> parents: 26442 diff changeset	486
38379 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38366 diff changeset	487 progress.complete()
52718 061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	488 _report_transferred(repo, start, bytecount)
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	489
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	490
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	491 def readbundle1header(fp) -> tuple[int, int, Set[bytes]]:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	492 compression = fp.read(2)
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	493 if compression != b'UN':
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	494 raise error.Abort(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	495 _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	496 b'only uncompressed stream clone bundles are '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	497 b'supported; got %s'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	498 )
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	499 % compression
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	500 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	501
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	502 filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	503 requireslen = struct.unpack(b'>H', fp.read(2))[0]
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	504 requires = fp.read(requireslen)
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	505
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	506 if not requires.endswith(b'\0'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	507 raise error.Abort(
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	508 _(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	509 b'malformed stream clone bundle: '
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	510 b'requirements not properly encoded'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	511 )
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	512 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	513
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	514 requirements = set(requires.rstrip(b'\0').split(b','))
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	515
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	516 return filecount, bytecount, requirements
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	517
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	518
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	519 def applybundlev1(repo, fp) -> None:
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	520 """Apply the content from a stream clone bundle version 1.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	521
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	522 We assume the 4 byte header has been read and validated and the file handle
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	523 is at the 2 byte compression identifier.
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	524 """
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	525 if len(repo):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	526 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	527 _(b'cannot apply stream clone bundle on non-empty repo')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	528 )
27882 319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	529
319b0bf6ecc9 streamclone: extract code for reading header fields Gregory Szorc <gregory.szorc@gmail.com> parents: 27870 diff changeset	530 filecount, bytecount, requirements = readbundle1header(fp)
48359 6d2ddea0721a stream-clone: filter possible missing requirements using all supported one Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	531 missingreqs = requirements - repo.supported
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	532 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	533 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	534 _(b'unable to apply stream clone: unsupported format: %s')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	535 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	536 )
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	537
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	538 consumev1(repo, fp, filecount, bytecount)
48707 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48695 diff changeset	539 nodemap.post_stream_cleanup(repo)
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	540
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	541
49037 642e31cb55f0 py3: use class X: instead of class X(object): Gregory Szorc <gregory.szorc@gmail.com> parents: 48966 diff changeset	542 class streamcloneapplier:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	543 """Class to manage applying streaming clone bundles.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	544
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	545 We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	546 readers to perform bundle type-specific functionality.
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	547 """
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	548
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	549 def __init__(self, fh) -> None:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	550 self._fh = fh
bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	551
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	552 def apply(self, repo) -> None:
26755 bb0b955d050d streamclone: support for producing and consuming stream clone bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	553 return applybundlev1(repo, self._fh)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	554
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	555
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	556 # type of file to stream
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	557 _fileappend = 0 # append only file
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	558 _filefull = 1 # full snapshot file
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	559
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	560 # Source of the file
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	561 _srcstore = b's' # store (svfs)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	562 _srccache = b'c' # cache (cache)
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	563
51762 ca7bde5dbafb black: format the codebase with 23.3.0 Rapha?l Gom?s <rgomes@octobus.net> parents: 51759 diff changeset	564
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	565 # This is it's own function so extensions can override it.
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	566 def _walkstreamfullstorefiles(repo) -> list[bytes]:
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	567 """list snapshot file from the store"""
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	568 fnames = []
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	569 if not repo.publishing():
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	570 fnames.append(b'phaseroots')
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	571 return fnames
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	572
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	573
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	574 def _filterfull(entry, copy, vfsmap):
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	575 """actually copy the snapshot files"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	576 src, name, ftype, data = entry
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	577 if ftype != _filefull:
56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	578 return entry
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	579 return (src, name, ftype, copy(vfsmap[src].join(name)))
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	580
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	581
52043 e308439339e2 stream: rename TempCopyManager to VolatileManager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51901 diff changeset	582 class VolatileManager:
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	583 """Manage temporary backups of volatile files during stream clone.
50660 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	584
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	585 This class will keep open file handles for the volatile files, writing the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	586 smaller ones on disk if the number of open file handles grow too much.
50660 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	587
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	588 This should be used as a Python context, the file handles and copies will
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	589 be discarded when exiting the context.
49222 4ff4e23de7df clone: use better names for temp files Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 49037 diff changeset	590
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	591 The preservation can be done by calling the object on the real path
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	592 (encoded full path).
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	593
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	594 Valid filehandles for any file should be retrieved by calling `open(path)`.
50660 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	595 """
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	596
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	597 # arbitrarily picked as "it seemed fine" and much higher than the current
52154 e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52046 diff changeset	598 # usage. The Windows value of 2 is actually 1 file open at a time, due to
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52046 diff changeset	599 # the `flush_count = self.MAX_OPEN // 2` and `self.MAX_OPEN - 1` threshold
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52046 diff changeset	600 # for flushing to disk in __call__().
e4b242f9d4d9 streamclone: disable the volatile file open handle optimization on Windows Matt Harbison <matt_harbison@yahoo.com> parents: 52046 diff changeset	601 MAX_OPEN = 2 if pycompat.iswindows else 100
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	602
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	603 def __init__(self) -> None:
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	604 self._counter = 0
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	605 self._volatile_fps = None
50660 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	606 self._copies = None
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	607 self._dst_dir = None
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	608
50660 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	609 def __enter__(self):
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	610 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	611 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	612 self._volatile_fps = {}
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	613 self._counter += 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	614 return self
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	615
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	616 def __exit__(self, args, *kwars):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	617 """discard all backups"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	618 self._counter -= 1
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	619 if self._counter == 0:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	620 for _size, fp in self._volatile_fps.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	621 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	622 self._volatile_fps = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	623 if self._copies is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	624 for tmp in self._copies.values():
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	625 util.tryunlink(tmp)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	626 util.tryrmdir(self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	627 self._copies = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	628 self._dst_dir = None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	629 assert self._volatile_fps is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	630 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	631 assert self._dst_dir is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	632
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	633 def _init_tmp_copies(self) -> None:
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	634 """prepare a temporary directory to save volatile files
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	635
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	636 This will be used as backup if we have too many files open"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	637 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	638 assert self._copies is None
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	639 assert self._dst_dir is None
50660 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	640 self._copies = {}
8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	641 self._dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	642
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	643 def _flush_some_on_disk(self) -> None:
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	644 """move some of the open files to tempory files on disk"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	645 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	646 self._init_tmp_copies()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	647 flush_count = self.MAX_OPEN // 2
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	648 for src, (size, fp) in sorted(self._volatile_fps.items())[:flush_count]:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	649 prefix = os.path.basename(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	650 fd, dst = pycompat.mkstemp(prefix=prefix, dir=self._dst_dir)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	651 self._copies[src] = dst
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	652 os.close(fd)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	653 # we no longer hardlink, but on the other hand we rarely do this,
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	654 # and we do it for the smallest file only and not at all in the
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	655 # common case.
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	656 with open(dst, 'wb') as bck:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	657 fp.seek(0)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	658 bck.write(fp.read())
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	659 del self._volatile_fps[src]
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	660 fp.close()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	661
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	662 def _keep_one(self, src: bytes) -> int:
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	663 """preserve an open file handle for a given path"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	664 # store the file quickly to ensure we close it if any error happens
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	665 _, fp = self._volatile_fps[src] = (None, open(src, 'rb'))
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	666 fp.seek(0, os.SEEK_END)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	667 size = fp.tell()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	668 self._volatile_fps[src] = (size, fp)
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	669 return size
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	670
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	671 def __call__(self, src: bytes) -> None:
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	672 """preserve the volatile file at src"""
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	673 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	674 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	675 self._flush_some_on_disk()
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	676 self._keep_one(src)
50660 8c7b04e69894 stream-clone: introduce a richer TempCopyManager object Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50558 diff changeset	677
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	678 def try_keep(self, src: bytes) -> Optional[int]:
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	679 """record a volatile file and returns it size
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	680
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	681 return None if the file does not exists.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	682
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	683 Used for cache file that are not lock protected.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	684 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	685 assert 0 < self._counter
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	686 if len(self._volatile_fps) >= (self.MAX_OPEN - 1):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	687 self._flush_some_on_disk()
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	688 try:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	689 return self._keep_one(src)
52665 24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError` Matt Harbison <matt_harbison@yahoo.com> parents: 52391 diff changeset	690 except OSError as err:
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	691 if err.errno not in (errno.ENOENT, errno.EPERM):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	692 raise
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	693 return None
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	694
52044 0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52043 diff changeset	695 @contextlib.contextmanager
0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52043 diff changeset	696 def open(self, src):
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	697 assert 0 < self._counter
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	698 entry = self._volatile_fps.get(src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	699 if entry is not None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	700 _size, fp = entry
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	701 fp.seek(0)
52044 0ad269e24075 stream: open volatile file through the manager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52043 diff changeset	702 yield fp
52046 a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	703 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	704 if self._copies is None:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	705 actual_path = src
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	706 else:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	707 actual_path = self._copies.get(src, src)
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	708 with open(actual_path, 'rb') as fp:
a47f09da8bd1 stream: prefer keeping an open file handle to volatile file instead of copy Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52045 diff changeset	709 yield fp
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	710
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	711
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	712 def _makemap(repo):
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	713 """make a (src -> vfs) map for the repo"""
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	714 vfsmap = {
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	715 _srcstore: repo.svfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	716 _srccache: repo.cachevfs,
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	717 }
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	718 # we keep repo.vfs out of the on purpose, ther are too many danger there
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	719 # (eg: .hg/hgrc)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	720 assert repo.vfs not in vfsmap.values()
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	721
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	722 return vfsmap
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	723
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	724
50665 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	725 def _emit2(repo, entries):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	726 """actually emit the stream bundle"""
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	727 vfsmap = _makemap(repo)
47050 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	728 # we keep repo.vfs out of the on purpose, ther are too many danger there
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	729 # (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	730 #
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	731 # this assert is duplicated (from _makemap) as author might think this is
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	732 # fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	733 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	734 raise error.ProgrammingError(
52714 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52713 diff changeset	735 'repo.vfs must not be added to vfsmap for security reasons'
47050 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	736 )
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	737
50665 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	738 # translate the vfs one
3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	739 entries = [(vfs_key, vfsmap[vfs_key], e) for (vfs_key, e) in entries]
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	740 _test_sync_point_walk_1_2(repo)
50665 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	741
50670 9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50669 diff changeset	742 max_linkrev = len(repo)
50665 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	743 file_count = totalfilesize = 0
52388 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	744 with VolatileManager() as volatiles:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	745 # make sure we preserve volatile files
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	746 with util.nogc():
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	747 # record the expected size of every file
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	748 for k, vfs, e in entries:
52389 a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52388 diff changeset	749 e.preserve_volatiles(vfs, volatiles)
52388 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	750 for f in e.files():
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	751 file_count += 1
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	752 totalfilesize += f.file_size(vfs)
50665 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	753
52388 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	754 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	755 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	756 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	757 progress.update(0)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	758 with progress:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	759 # the first yield release the lock on the repository
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	760 yield file_count, totalfilesize
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	761 totalbytecount = 0
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	762
52388 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	763 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	764 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	765 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	766 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	767 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	768 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	769 preserve_file_count=True,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	770 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	771 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	772 yield src
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	773 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	774 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	775 yield name
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	776 bytecount = 0
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	777 for chunk in stream:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	778 bytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	779 totalbytecount += len(chunk)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	780 progress.update(totalbytecount)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	781 yield chunk
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	782 if bytecount != size:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	783 # Would most likely be caused by a race due to `hg
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	784 # strip` or a revlog split
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	785 msg = _(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	786 b'clone could only read %d bytes from %s, but '
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	787 b'expected %d bytes'
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	788 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	789 raise error.Abort(msg % (bytecount, name, size))
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	790
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	791
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	792 def _emit3(repo, entries) -> Iterator[bytes \| None]:
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	793 """actually emit the stream bundle (v3)"""
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	794 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	795 # we keep repo.vfs out of the map on purpose, ther are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	796 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	797 #
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	798 # this assert is duplicated (from _makemap) as authors might think this is
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	799 # fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	800 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	801 raise error.ProgrammingError(
52714 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52713 diff changeset	802 'repo.vfs must not be added to vfsmap for security reasons'
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	803 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	804
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	805 # translate the vfs once
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	806 # we only turn this into a list for the `_test_sync`, this is not ideal
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	807 base_entries = list(entries)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	808 _test_sync_point_walk_1_2(repo)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	809 entries = []
52388 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	810 with VolatileManager() as volatiles:
52043 e308439339e2 stream: rename TempCopyManager to VolatileManager Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51901 diff changeset	811 # make sure we preserve volatile files
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	812 for vfs_key, e in base_entries:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	813 vfs = vfsmap[vfs_key]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	814 any_files = True
51590 49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51589 diff changeset	815 if e.maybe_volatile:
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	816 any_files = False
52389 a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52388 diff changeset	817 e.preserve_volatiles(vfs, volatiles)
51590 49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51589 diff changeset	818 for f in e.files():
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51589 diff changeset	819 if f.is_volatile:
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51589 diff changeset	820 # record the expected size under lock
49faa72b994e streamclone: stop listing files for entries that have no volatile files Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51589 diff changeset	821 f.file_size(vfs)
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	822 any_files = True
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	823 if any_files:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	824 entries.append((vfs_key, vfsmap[vfs_key], e))
52388 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	825
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	826 total_entry_count = len(entries)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	827
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	828 max_linkrev = len(repo)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	829 progress = repo.ui.makeprogress(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	830 _(b'bundle'),
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	831 total=total_entry_count,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	832 unit=_(b'entry'),
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	833 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	834 progress.update(0)
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	835 # the first yield release the lock on the repository
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	836 yield None
52388 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	837 with progress:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	838 yield util.uvarintencode(total_entry_count)
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	839
52388 46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	840 for src, vfs, e in entries:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	841 entry_streams = e.get_streams(
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	842 repo=repo,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	843 vfs=vfs,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	844 volatiles=volatiles,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	845 max_changeset=max_linkrev,
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	846 )
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	847 yield util.uvarintencode(len(entry_streams))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	848 for name, stream, size in entry_streams:
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	849 yield src
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	850 yield util.uvarintencode(len(name))
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	851 yield util.uvarintencode(size)
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	852 yield name
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	853 yield from stream
46574e588017 stream: start the volatile manager sooner during emission Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52154 diff changeset	854 progress.increment()
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	855
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	856
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	857 def _test_sync_point_walk_1_2(repo):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	858 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	859
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	860 Triggered after gather entry, but before starting to process/preserve them
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	861 under lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	862
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	863 (on v1 is triggered before the actual walk start)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	864 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	865
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	866
52390 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52389 diff changeset	867 def _test_sync_point_walk_3(repo):
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	868 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	869
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	870 Triggered right before releasing the lock, but after computing what need
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	871 needed to compute under lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	872 """
46986 faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	873
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	874
52390 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52389 diff changeset	875 def _test_sync_point_walk_4(repo):
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	876 """a function for synchronisation during tests
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	877
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	878 Triggered right after releasing the lock.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	879 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	880
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	881
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	882 # not really a StoreEntry, but close enough
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	883 class CacheEntry(store.SimpleStoreEntry):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	884 """Represent an entry for Cache files
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	885
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	886 It has special logic to preserve cache file early and accept optional
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	887 presence.
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	888
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	889
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	890 (Yes... this is not really a StoreEntry, but close enough. We could have a
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	891 BaseEntry base class, bbut the store one would be identical)
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	892 """
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	893
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	894 def __init__(self, entry_path) -> None:
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	895 super().__init__(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	896 entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	897 # we will directly deal with that in `setup_cache_file`
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	898 is_volatile=True,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	899 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	900
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	901 def preserve_volatiles(self, vfs, volatiles) -> None:
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	902 self._file_size = volatiles.try_keep(vfs.join(self._entry_path))
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	903 if self._file_size is None:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	904 self._files = []
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	905 else:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	906 assert self._is_volatile
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	907 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	908 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	909 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	910 file_size=self._file_size,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	911 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	912 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	913 ]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	914
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	915 def files(self) -> list[store.StoreFile]:
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	916 if self._files is None:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	917 self._files = [
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	918 CacheFile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	919 unencoded_path=self._entry_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	920 is_volatile=self._is_volatile,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	921 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	922 ]
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	923 return super().files()
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	924
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	925
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	926 class CacheFile(store.StoreFile):
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	927 # inform the "copy/hardlink" version that this file might be missing
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	928 # without consequences.
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	929 optional: bool = True
46986 faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	930
faa43f09ad98 streamclone: remove sleep based "synchronisation" in tests Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46895 diff changeset	931
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	932 def _entries_walk(repo, includes, excludes, includeobsmarkers: bool):
50549 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	933 """emit a seris of files information useful to clone a repo
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	934
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	935 return (vfs-key, entry) iterator
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	936
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	937 Where `entry` is StoreEntry. (used even for cache entries)
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	938 """
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	939 assert repo._currentlock(repo._lockref) is not None
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	940
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	941 matcher = None
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	942 if includes or excludes:
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	943 matcher = narrowspec.match(repo.root, includes, excludes)
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	944
06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	945 phase = not repo.publishing()
51588 463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	946 # Python is getting crazy at all the small container we creates, disabling
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	947 # the gc while we do so helps performance a lot.
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	948 with util.nogc():
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	949 entries = _walkstreamfiles(
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	950 repo,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	951 matcher,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	952 phase=phase,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	953 obsolescence=includeobsmarkers,
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	954 )
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	955 for entry in entries:
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	956 yield (_srcstore, entry)
50549 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	957
51588 463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	958 for name in cacheutil.cachetocopy(repo):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	959 if repo.cachevfs.exists(name):
463e63aa547c stream-clone: disable gc for `_entries_walk` duration Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51586 diff changeset	960 # not really a StoreEntry, but close enough
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	961 yield (_srccache, CacheEntry(entry_path=name))
50550 43ed1f12b00a stream-clone: yield cache entry in `_entries_walk` too Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50549 diff changeset	962
50549 06d580b8f432 stream-clone: introduce a _entries_walk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50548 diff changeset	963
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	964 def generatev2(repo, includes, excludes, includeobsmarkers: bool):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	965 """Emit content for version 2 of a streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	966
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	967 the data stream consists the following entries:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	968 1) A char representing the file destination (eg: store or cache)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	969 2) A varint containing the length of the filename
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	970 3) A varint containing the length of file data
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	971 4) N bytes containing the filename (the internal, store-agnostic form)
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	972 5) N bytes containing the file data
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	973
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	974 Returns a 3-tuple of (file count, file size, data iterator).
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	975 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	976
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	977 with repo.lock():
47450 2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	978 repo.ui.debug(b'scanning\n')
40339 f0e8f27768eb streamclone: pass narrowing related info in _walkstreamfiles() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40338 diff changeset	979
50665 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	980 entries = _entries_walk(
47450 2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	981 repo,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	982 includes=includes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	983 excludes=excludes,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	984 includeobsmarkers=includeobsmarkers,
2f4ca4807033 streamingclone: extract the scanning part from the generation part Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	985 )
50665 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	986 chunks = _emit2(repo, entries)
35765 56c30b31afbe streamclone: add support for cloning non append-only file Boris Feld <boris.feld@octobus.net> parents: 35757 diff changeset	987 first = next(chunks)
50665 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	988 file_count, total_file_size = first
52390 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52389 diff changeset	989 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52389 diff changeset	990 _test_sync_point_walk_4(repo)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	991
50665 3416b46320dc stream-clone: directly use `_entries_walk` to generate stream-v2 Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50664 diff changeset	992 return file_count, total_file_size, chunks
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	993
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	994
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	995 def generatev3(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	996 repo, includes, excludes, includeobsmarkers: bool
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	997 ) -> Iterator[bytes \| None]:
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	998 """Emit content for version 3 of a streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	999
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1000 the data stream consists the following:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1001 1) A varint E containing the number of entries (can be 0), then E entries follow
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1002 2) For each entry:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1003 2.1) The number of files in this entry (can be 0, but typically 1 or 2)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1004 2.2) For each file:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1005 2.2.1) A char representing the file destination (eg: store or cache)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1006 2.2.2) A varint N containing the length of the filename
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1007 2.2.3) A varint M containing the length of file data
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1008 2.2.4) N bytes containing the filename (the internal, store-agnostic form)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1009 2.2.5) M bytes containing the file data
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1010
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1011 Returns the data iterator.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1012
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1013 XXX This format is experimental and subject to change. Here is a
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1014 XXX non-exhaustive list of things this format could do or change:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1015
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1016 - making it easier to write files in parallel
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1017 - holding the lock for a shorter time
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1018 - improving progress information
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1019 - ways to adjust the number of expected entries/files ?
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1020 """
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1021
51589 6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51588 diff changeset	1022 # Python is getting crazy at all the small container we creates while
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51588 diff changeset	1023 # considering the files to preserve, disabling the gc while we do so helps
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51588 diff changeset	1024 # performance a lot.
6e4c8366c5ce stream-clone: disable gc for the initial section for the v3 format Pierre-Yves David <pierre-yves.david@octobus.net> parents: 51588 diff changeset	1025 with repo.lock(), util.nogc():
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1026 repo.ui.debug(b'scanning\n')
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1027
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1028 entries = _entries_walk(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1029 repo,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1030 includes=includes,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1031 excludes=excludes,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1032 includeobsmarkers=includeobsmarkers,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1033 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1034 chunks = _emit3(repo, list(entries))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1035 first = next(chunks)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1036 assert first is None
52390 11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52389 diff changeset	1037 _test_sync_point_walk_3(repo)
11484a19cd77 stream: rename all test hook point one number up Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52389 diff changeset	1038 _test_sync_point_walk_4(repo)
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1039
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1040 return chunks
50558 58adcabc295f stream-clone: introduce the notion of an experimental "v3" version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50557 diff changeset	1041
58adcabc295f stream-clone: introduce the notion of an experimental "v3" version Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50557 diff changeset	1042
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1043 @contextlib.contextmanager
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1044 def nested(*ctxs):
39773 97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	1045 this = ctxs[0]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	1046 rest = ctxs[1:]
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	1047 with this:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	1048 if rest:
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	1049 with nested(*rest):
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	1050 yield
97f2992c26f6 streamclone: reimplement nested context manager Augie Fackler <augie@google.com> parents: 39716 diff changeset	1051 else:
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1052 yield
5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1053
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1054
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	1055 def consumev2(repo, fp, filecount: int, filesize: int) -> None:
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1056 """Apply the contents from a version 2 streaming clone.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1057
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1058 Data is read from an object that only needs to provide a ``read(size)``
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1059 method.
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1060 """
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1061 with repo.lock():
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1062 repo.ui.status(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1063 _(b'%d files to transfer, %s of data\n')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1064 % (filecount, util.bytecount(filesize))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1065 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1066
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1067 start = util.timer()
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1068 progress = repo.ui.makeprogress(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1069 _(b'clone'), total=filesize, unit=_(b'bytes')
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1070 )
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	1071 progress.update(0)
52719 6feb3b3029b5 stream: consistently close progress before reporting time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52718 diff changeset	1072 byte_count = 0
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1073
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1074 vfsmap = _makemap(repo)
47050 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1075 # we keep repo.vfs out of the on purpose, ther are too many danger
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1076 # there (eg: .hg/hgrc),
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1077 #
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1078 # this assert is duplicated (from _makemap) as author might think this
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1079 # is fine, while this is really not fine.
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1080 if repo.vfs in vfsmap.values():
65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1081 raise error.ProgrammingError(
52714 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52713 diff changeset	1082 'repo.vfs must not be added to vfsmap for security reasons'
47050 65c519661991 stream: double check that self.vfs is not in the vfsmap Pierre-Yves David <pierre-yves.david@octobus.net> parents: 46989 diff changeset	1083 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1084
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1085 with repo.transaction(b'clone'):
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1086 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1087 with nested(*ctxs):
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1088 for i in range(filecount):
35803 3ad3aaeb1134 streamclone: use readexactly when reading stream v2 Boris Feld <boris.feld@octobus.net> parents: 35802 diff changeset	1089 src = util.readexactly(fp, 1)
35767 5f5fb279fd39 streamclone: also stream caches to the client Boris Feld <boris.feld@octobus.net> parents: 35765 diff changeset	1090 vfs = vfsmap[src]
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1091 namelen = util.uvarintdecodestream(fp)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1092 datalen = util.uvarintdecodestream(fp)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1093
35803 3ad3aaeb1134 streamclone: use readexactly when reading stream v2 Boris Feld <boris.feld@octobus.net> parents: 35802 diff changeset	1094 name = util.readexactly(fp, namelen)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1095
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1096 if repo.ui.debugflag:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1097 repo.ui.debug(
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1098 b'adding [%s] %s (%s)\n'
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1099 % (src, name, util.bytecount(datalen))
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1100 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1101
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1102 with vfs(name, b'w') as ofp:
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1103 for chunk in util.filechunkiter(fp, limit=datalen):
52719 6feb3b3029b5 stream: consistently close progress before reporting time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52718 diff changeset	1104 byte_count += len(chunk)
38355 e59eaf51cc0d streamclone: use progress helper Martin von Zweigbergk <martinvonz@google.com> parents: 38197 diff changeset	1105 progress.increment(step=len(chunk))
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1106 ofp.write(chunk)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1107
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1108 # force @filecache properties to be reloaded from
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1109 # streamclone-ed file at next access
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1110 repo.invalidate(clearfilecache=True)
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1111
38379 ef692614e601 progress: hide update(None) in a new complete() method Martin von Zweigbergk <martinvonz@google.com> parents: 38366 diff changeset	1112 progress.complete()
52719 6feb3b3029b5 stream: consistently close progress before reporting time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52718 diff changeset	1113 _report_transferred(repo, start, byte_count)
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1114
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1115
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	1116 def consumev3(repo, fp) -> None:
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1117 """Apply the contents from a version 3 streaming clone.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1118
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1119 Data is read from an object that only needs to provide a ``read(size)``
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1120 method.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1121 """
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1122 with repo.lock():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1123 start = util.timer()
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1124
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1125 entrycount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1126 repo.ui.status(_(b'%d entries to transfer\n') % (entrycount))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1127
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1128 progress = repo.ui.makeprogress(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1129 _(b'clone'),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1130 total=entrycount,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1131 unit=_(b'entries'),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1132 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1133 progress.update(0)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1134 bytes_transferred = 0
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1135
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1136 vfsmap = _makemap(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1137 # we keep repo.vfs out of the on purpose, there are too many dangers
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1138 # there (eg: .hg/hgrc),
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1139 #
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1140 # this assert is duplicated (from _makemap) as authors might think this
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1141 # is fine, while this is really not fine.
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1142 if repo.vfs in vfsmap.values():
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1143 raise error.ProgrammingError(
52714 10e7adbffa8c streamclone: unbyteify string args to builtin Error classes Matt Harbison <matt_harbison@yahoo.com> parents: 52713 diff changeset	1144 'repo.vfs must not be added to vfsmap for security reasons'
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1145 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1146
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1147 with repo.transaction(b'clone'):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1148 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1149 with nested(*ctxs):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1150 for i in range(entrycount):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1151 filecount = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1152 if filecount == 0:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1153 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1154 repo.ui.debug(b'entry with no files [%d]\n' % (i))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1155 for i in range(filecount):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1156 src = util.readexactly(fp, 1)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1157 vfs = vfsmap[src]
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1158 namelen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1159 datalen = util.uvarintdecodestream(fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1160
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1161 name = util.readexactly(fp, namelen)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1162
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1163 if repo.ui.debugflag:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1164 msg = b'adding [%s] %s (%s)\n'
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1165 msg %= (src, name, util.bytecount(datalen))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1166 repo.ui.debug(msg)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1167 bytes_transferred += datalen
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1168
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1169 with vfs(name, b'w') as ofp:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1170 for chunk in util.filechunkiter(fp, limit=datalen):
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1171 ofp.write(chunk)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1172 progress.increment(step=1)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1173
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1174 # force @filecache properties to be reloaded from
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1175 # streamclone-ed file at next access
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1176 repo.invalidate(clearfilecache=True)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1177
52719 6feb3b3029b5 stream: consistently close progress before reporting time Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52718 diff changeset	1178 progress.complete()
52718 061bfd699a56 stream: use an utility function to report transferred bytes Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52714 diff changeset	1179 _report_transferred(repo, start, bytes_transferred)
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1180
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1181
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	1182 def applybundlev2(
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	1183 repo, fp, filecount: int, filesize: int, requirements: Iterable[bytes]
f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	1184 ) -> None:
39716 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	1185 from . import localrepo
b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	1186
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1187 missingreqs = [r for r in requirements if r not in repo.supported]
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1188 if missingreqs:
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1189 raise error.Abort(
43117 8ff1ecfadcd1 cleanup: join string literals that are already on one line Martin von Zweigbergk <martinvonz@google.com> parents: 43085 diff changeset	1190 _(b'unable to apply stream clone: unsupported format: %s')
43077 687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals Augie Fackler <augie@google.com> parents: 43076 diff changeset	1191 % b', '.join(sorted(missingreqs))
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1192 )
35756 cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1193
cfdccd560b66 streamclone: define first iteration of version 2 of stream format Boris Feld <boris.feld@octobus.net> parents: 35495 diff changeset	1194 consumev2(repo, fp, filecount, filesize)
35804 2d3e486d09d0 streamclone: move requirement update into consumev2 Boris Feld <boris.feld@octobus.net> parents: 35803 diff changeset	1195
48618 739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	1196 repo.requirements = new_stream_clone_requirements(
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	1197 repo.requirements,
739f2ca3aa3f stream-clone: factor computation of new clone requirement out Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47919 diff changeset	1198 requirements,
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1199 )
39716 b10d145837bc localrepo: extract resolving of opener options to standalone functions Gregory Szorc <gregory.szorc@gmail.com> parents: 38823 diff changeset	1200 repo.svfs.options = localrepo.resolvestorevfsoptions(
43076 2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1201 repo.ui, repo.requirements, repo.features
2372284d9457 formatting: blacken the codebase Augie Fackler <augie@google.com> parents: 42823 diff changeset	1202 )
45106 a03c177a4679 scmutil: add writereporequirements() and route requires writing through it Pulkit Goyal <7895pulkit@gmail.com> parents: 43117 diff changeset	1203 scmutil.writereporequirements(repo)
48707 de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 48695 diff changeset	1204 nodemap.post_stream_cleanup(repo)
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1205
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1206
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	1207 def applybundlev3(repo, fp, requirements: Iterable[bytes]) -> None:
50706 0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1208 from . import localrepo
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1209
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1210 missingreqs = [r for r in requirements if r not in repo.supported]
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1211 if missingreqs:
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1212 msg = _(b'unable to apply stream clone: unsupported format: %s')
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1213 msg %= b', '.join(sorted(missingreqs))
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1214 raise error.Abort(msg)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1215
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1216 consumev3(repo, fp)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1217
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1218 repo.requirements = new_stream_clone_requirements(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1219 repo.requirements,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1220 requirements,
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1221 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1222 repo.svfs.options = localrepo.resolvestorevfsoptions(
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1223 repo.ui, repo.requirements, repo.features
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1224 )
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1225 scmutil.writereporequirements(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1226 nodemap.post_stream_cleanup(repo)
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1227
0452af304808 stream-clone: add a v3 version of the protocol Arseniy Alekseyev <aalekseyev@janestreet.com> parents: 50675 diff changeset	1228
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	1229 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress) -> bool:
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1230 hardlink = [True]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1231
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1232 def copy_used():
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1233 hardlink[0] = False
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1234 progress.topic = _(b'copying')
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1235
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1236 for k, path, optional in entries:
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1237 src_vfs = src_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1238 dst_vfs = dst_vfs_map[k]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1239 src_path = src_vfs.join(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1240 dst_path = dst_vfs.join(path)
47863 132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47787 diff changeset	1241 # We cannot use dirname and makedirs of dst_vfs here because the store
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47787 diff changeset	1242 # encoding confuses them. See issue 6581 for details.
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47787 diff changeset	1243 dirname = os.path.dirname(dst_path)
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47787 diff changeset	1244 if not os.path.exists(dirname):
132525ead0db clone: properly create target directories during local clone (issue6581) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47787 diff changeset	1245 util.makedirs(dirname)
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1246 dst_vfs.register_file(path)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1247 # XXX we could use the #nb_bytes argument.
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1248 try:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1249 util.copyfile(
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1250 src_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1251 dst_path,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1252 hardlink=hardlink[0],
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1253 no_hardlink_cb=copy_used,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1254 check_fs_hardlink=False,
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1255 )
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1256 except FileNotFoundError:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1257 if not optional:
3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1258 raise
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1259 progress.increment()
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1260 return hardlink[0]
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1261
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1262
52713 f5471af96a52 typing: add trivial type annotations to `mercurial/streamclone.py` Matt Harbison <matt_harbison@yahoo.com> parents: 52709 diff changeset	1263 def local_copy(src_repo, dest_repo) -> None:
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1264 """copy all content from one local repository to another
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1265
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1266 This is useful for local clone"""
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1267 src_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1268 r
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1269 for r in src_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1270 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1271 }
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1272 dest_store_requirements = {
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1273 r
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1274 for r in dest_repo.requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1275 if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1276 }
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1277 assert src_store_requirements == dest_store_requirements
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1278
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1279 with dest_repo.lock():
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1280 with src_repo.lock():
47454 d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1281 # bookmark is not integrated to the streaming as it might use the
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1282 # `repo.vfs` and they are too many sentitive data accessible
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1283 # through `repo.vfs` to expose it to streaming clone.
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1284 src_book_vfs = bookmarks.bookmarksvfs(src_repo)
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1285 srcbookmarks = src_book_vfs.join(b'bookmarks')
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1286 bm_count = 0
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1287 if os.path.exists(srcbookmarks):
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1288 bm_count = 1
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1289
50663 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50660 diff changeset	1290 entries = _entries_walk(
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1291 src_repo,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1292 includes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1293 excludes=None,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1294 includeobsmarkers=True,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1295 )
50663 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50660 diff changeset	1296 entries = list(entries)
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1297 src_vfs_map = _makemap(src_repo)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1298 dest_vfs_map = _makemap(dest_repo)
50663 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50660 diff changeset	1299 total_files = sum(len(e[1].files()) for e in entries) + bm_count
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1300 progress = src_repo.ui.makeprogress(
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1301 topic=_(b'linking'),
50663 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50660 diff changeset	1302 total=total_files,
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1303 unit=_(b'files'),
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1304 )
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1305 # copy files
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1306 #
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1307 # We could copy the full file while the source repository is locked
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1308 # and the other one without the lock. However, in the linking case,
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1309 # this would also requires checks that nobody is appending any data
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1310 # to the files while we do the clone, so this is not done yet. We
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1311 # could do this blindly when copying files.
50663 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50660 diff changeset	1312 files = [
52391 3f0cf7bb3086 stream: preserve volatile cache early Pierre-Yves David <pierre-yves.david@octobus.net> parents: 52390 diff changeset	1313 (vfs_key, f.unencoded_path, f.optional)
50663 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50660 diff changeset	1314 for vfs_key, e in entries
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50660 diff changeset	1315 for f in e.files()
f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50660 diff changeset	1316 ]
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1317 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1318
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1319 # copy bookmarks over
47454 d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1320 if bm_count:
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1321 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
d370256636fe clone: also report the bookmark file as copied Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47453 diff changeset	1322 dstbookmarks = dst_book_vfs.join(b'bookmarks')
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1323 util.copyfile(srcbookmarks, dstbookmarks)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1324 progress.complete()
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1325 if hardlink:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1326 msg = b'linked %d files\n'
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1327 else:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1328 msg = b'copied %d files\n'
50663 f2ae815ae34c local-clone: perform the hardlink/copy based from _entries_walk returns Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50660 diff changeset	1329 src_repo.ui.debug(msg % total_files)
47453 377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1330
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1331 with dest_repo.transaction(b"localclone") as tr:
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1332 dest_repo.store.write(tr)
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1333
377d8fc20e34 clone: reuse the stream clone logic for local clone Pierre-Yves David <pierre-yves.david@octobus.net> parents: 47450 diff changeset	1334 # clean up transaction file as they do not make sense
50288 d89eecf9605e undo-files: no longer pass the `repo` to `cleanup_undo_files` Pierre-Yves David <pierre-yves.david@octobus.net> parents: 50286 diff changeset	1335 transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/streamclone.py @ 52719:6feb3b3029b5