mercurial/store.py
author Matt Harbison <matt_harbison@yahoo.com>
Sun, 05 Jan 2025 21:03:17 -0500
changeset 52640 24ee91ba9aa8
parent 52639 9db77d46de79
child 52643 5cc8deb96b48
permissions -rw-r--r--
pyupgrade: drop usage of py3 aliases for `OSError` These were different classes in py2, but now a handful of error classes are just an alias of `OSError`, like `IOError`, `EnvironmentError`, `WindowsError`, etc. This is the result of running a hacked version of `pyupgrade` 3.19.1[1] $ hg files -0 'relglob:**.py' | xargs -0 \ pyupgrade --py38-plus --keep-percent-format --keep-mock --keep-runtime-typing The hack is because it doesn't have command line switches to disable most changes, so it makes tons of unrelated changes all at once. The hack is to 1) patch `pyupgrade._main._fix_tokens()` to immediately return its content arg 2) change `pyupgrade._data.register_decorator()` to only register the function if it's from the fixer we're interested in: if func.__module__ in ( "pyupgrade._plugins.exceptions", ): FUNCS[tp].append(func) return func [1] https://github.com/asottile/pyupgrade
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
50677
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
     1
# store.py - repository store handling for Mercurial)
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
     2
#
46819
d4ba4d51f85f contributor: change mentions of mpm to olivia
Rapha?l Gom?s <rgomes@octobus.net>
parents: 46780
diff changeset
     3
# Copyright 2008 Olivia Mackall <olivia@selenic.com>
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
     4
#
8225
46293a0c7e9f updated license to be explicit about GPL version 2
Martin Geisler <mg@lazybytes.net>
parents: 8210
diff changeset
     5
# This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9133
diff changeset
     6
# GNU General Public License version 2 or any later version.
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
     7
51859
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51822
diff changeset
     8
from __future__ import annotations
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 51822
diff changeset
     9
50476
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
    10
import collections
41978
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
    11
import functools
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
    12
import os
46990
0b569c75d180 store: exclude `undo.` nodemap's file from `walk`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
    13
import re
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
    14
import stat
51725
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51724
diff changeset
    15
import typing
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51724
diff changeset
    16
51878
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
    17
from typing import (
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
    18
    Generator,
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
    19
    List,
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
    20
    Optional,
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
    21
)
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
    22
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
    23
from .i18n import _
50468
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50466
diff changeset
    24
from .thirdparty import attr
51725
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51724
diff changeset
    25
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51724
diff changeset
    26
# Force pytype to use the non-vendored package
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51724
diff changeset
    27
if typing.TYPE_CHECKING:
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51724
diff changeset
    28
    # noinspection PyPackageRequirements
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51724
diff changeset
    29
    import attr
278af66e6595 typing: induce pytype to use the standard `attr` instead of the vendored copy
Matt Harbison <matt_harbison@yahoo.com>
parents: 51724
diff changeset
    30
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
    31
from .node import hex
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
    32
from .revlogutils.constants import (
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
    33
    INDEX_HEADER,
50677
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
    34
    KIND_CHANGELOG,
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
    35
    KIND_FILELOG,
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
    36
    KIND_MANIFESTLOG,
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
    37
)
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
    38
from . import (
42905
3df3b139a43d localrepo: push manifestlog and changelog construction code into store
Augie Fackler <augie@google.com>
parents: 42715
diff changeset
    39
    changelog,
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
    40
    error,
50624
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
    41
    filelog,
42905
3df3b139a43d localrepo: push manifestlog and changelog construction code into store
Augie Fackler <augie@google.com>
parents: 42715
diff changeset
    42
    manifest,
32372
df448de7cf3b parsers: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 31362
diff changeset
    43
    policy,
30077
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
    44
    pycompat,
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
    45
    revlog as revlogmod,
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
    46
    util,
31234
9b7a2ef4f27c vfs: use 'vfs' module directly in 'mercurial.store'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31219
diff changeset
    47
    vfs as vfsmod,
27480
509159675cdb store: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26778
diff changeset
    48
)
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 43506
diff changeset
    49
from .utils import hashutil
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
    50
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43106
diff changeset
    51
parsers = policy.importmod('parsers')
41978
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
    52
# how much bytes should be read from fncache in one read
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
    53
# It is done to prevent loading large fncache files into memory
51699
ca7bde5dbafb black: format the codebase with 23.3.0
Rapha?l Gom?s <rgomes@octobus.net>
parents: 51696
diff changeset
    54
fncache_chunksize = 10**6
32372
df448de7cf3b parsers: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 31362
diff changeset
    55
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
    56
51724
2e9e62242451 typing: disable some pytype errors in `mercurial.store`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
    57
def _match_tracked_entry(entry: "BaseStoreEntry", matcher):
40494
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
    58
    """parses a fncache entry and returns whether the entry is tracking a path
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
    59
    matched by matcher or not.
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
    60
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
    61
    If matcher is None, returns True"""
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
    62
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
    63
    if matcher is None:
9aeb9e2d28a7 store: introduce _matchtrackedpath() and use it to filter store files
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40340
diff changeset
    64
        return True
51724
2e9e62242451 typing: disable some pytype errors in `mercurial.store`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
    65
2e9e62242451 typing: disable some pytype errors in `mercurial.store`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
    66
    # TODO: make this safe for other entry types.  Currently, the various
2e9e62242451 typing: disable some pytype errors in `mercurial.store`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
    67
    #  store.data_entry generators only yield  RevlogStoreEntry, so the
2e9e62242451 typing: disable some pytype errors in `mercurial.store`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
    68
    #  attributes do exist on `entry`.
2e9e62242451 typing: disable some pytype errors in `mercurial.store`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
    69
    # pytype: disable=attribute-error
50495
1b776f25302f store: use the boolean property in `store`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50494
diff changeset
    70
    if entry.is_filelog:
50491
b4953fad744e store: do the revlog matching on entry directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50490
diff changeset
    71
        return matcher(entry.target_id)
50495
1b776f25302f store: use the boolean property in `store`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50494
diff changeset
    72
    elif entry.is_manifestlog:
50491
b4953fad744e store: do the revlog matching on entry directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50490
diff changeset
    73
        return matcher.visitdir(entry.target_id.rstrip(b'/'))
51724
2e9e62242451 typing: disable some pytype errors in `mercurial.store`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51699
diff changeset
    74
    # pytype: enable=attribute-error
50491
b4953fad744e store: do the revlog matching on entry directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50490
diff changeset
    75
    raise error.ProgrammingError(b"cannot process entry %r" % entry)
40624
66adfd58cb77 store: raise ProgrammingError if unable to decode a storage path
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40584
diff changeset
    76
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
    77
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
    78
# This avoids a collision between a file named foo and a dir named
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
    79
# foo.i or foo.d
17607
cc58dc47cb5e store: use fast C implementation of encodedir() if it's available
Adrian Buehlmann <adrian@cadifra.com>
parents: 17605
diff changeset
    80
def _encodedir(path):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
    81
    """
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
    82
    >>> _encodedir(b'data/foo.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
    83
    'data/foo.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
    84
    >>> _encodedir(b'data/foo.i/bla.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
    85
    'data/foo.i.hg/bla.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
    86
    >>> _encodedir(b'data/foo.i.hg/bla.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
    87
    'data/foo.i.hg.hg/bla.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
    88
    >>> _encodedir(b'data/foo.i\\ndata/foo.i/bla.i\\ndata/foo.i.hg/bla.i\\n')
17605
e9cc29be3305 store: add multiline doctest case for encodedir()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17604
diff changeset
    89
    'data/foo.i\\ndata/foo.i.hg/bla.i\\ndata/foo.i.hg.hg/bla.i\\n'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
    90
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
    91
    return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
    92
        path.replace(b".hg/", b".hg.hg/")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
    93
        .replace(b".i/", b".i.hg/")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
    94
        .replace(b".d/", b".d.hg/")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
    95
    )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
    96
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
    97
17607
cc58dc47cb5e store: use fast C implementation of encodedir() if it's available
Adrian Buehlmann <adrian@cadifra.com>
parents: 17605
diff changeset
    98
encodedir = getattr(parsers, 'encodedir', _encodedir)
cc58dc47cb5e store: use fast C implementation of encodedir() if it's available
Adrian Buehlmann <adrian@cadifra.com>
parents: 17605
diff changeset
    99
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   100
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
   101
def decodedir(path):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   102
    """
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   103
    >>> decodedir(b'data/foo.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   104
    'data/foo.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   105
    >>> decodedir(b'data/foo.i.hg/bla.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   106
    'data/foo.i/bla.i'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   107
    >>> decodedir(b'data/foo.i.hg.hg/bla.i')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   108
    'data/foo.i.hg/bla.i'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   109
    """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   110
    if b".hg/" not in path:
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
   111
        return path
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   112
    return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   113
        path.replace(b".d.hg/", b".d/")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   114
        .replace(b".i.hg/", b".i/")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   115
        .replace(b".hg.hg/", b".hg/")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   116
    )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   117
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
   118
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   119
def _reserved():
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   120
    """characters that are problematic for filesystems
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   121
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   122
    * ascii escapes (0..31)
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   123
    * ascii hi (126..255)
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   124
    * windows specials
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   125
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   126
    these characters will be escaped by encodefunctions
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   127
    """
52639
9db77d46de79 py3: drop redundant `u''` prefixes on string literals
Matt Harbison <matt_harbison@yahoo.com>
parents: 52359
diff changeset
   128
    winreserved = [ord(x) for x in '\\:*?"<>|']
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   129
    for x in range(32):
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   130
        yield x
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   131
    for x in range(126, 256):
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   132
        yield x
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   133
    for x in winreserved:
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   134
        yield x
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   135
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   136
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   137
def _buildencodefun():
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   138
    """
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   139
    >>> enc, dec = _buildencodefun()
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   140
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   141
    >>> enc(b'nothing/special.txt')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   142
    'nothing/special.txt'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   143
    >>> dec(b'nothing/special.txt')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   144
    'nothing/special.txt'
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   145
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   146
    >>> enc(b'HELLO')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   147
    '_h_e_l_l_o'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   148
    >>> dec(b'_h_e_l_l_o')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   149
    'HELLO'
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   150
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   151
    >>> enc(b'hello:world?')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   152
    'hello~3aworld~3f'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   153
    >>> dec(b'hello~3aworld~3f')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   154
    'hello:world?'
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   155
34136
414a3513c2bd doctest: do not embed non-ascii characters in docstring
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
   156
    >>> enc(b'the\\x07quick\\xADshot')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   157
    'the~07quick~adshot'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   158
    >>> dec(b'the~07quick~adshot')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   159
    'the\\x07quick\\xadshot'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   160
    """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   161
    e = b'_'
31253
64596338ba10 py3: factor out bytechr() function
Yuya Nishihara <yuya@tcha.org>
parents: 31234
diff changeset
   162
    xchr = pycompat.bytechr
64596338ba10 py3: factor out bytechr() function
Yuya Nishihara <yuya@tcha.org>
parents: 31234
diff changeset
   163
    asciistr = list(map(xchr, range(127)))
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   164
    capitals = list(range(ord(b"A"), ord(b"Z") + 1))
30077
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
   165
44452
9d2b2df2c2ba cleanup: run pyupgrade on our source tree to clean up varying things
Augie Fackler <augie@google.com>
parents: 44060
diff changeset
   166
    cmap = {x: x for x in asciistr}
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   167
    for x in _reserved():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   168
        cmap[xchr(x)] = b"~%02x" % x
30077
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
   169
    for x in capitals + [ord(e)]:
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
   170
        cmap[xchr(x)] = e + xchr(x).lower()
8f42d8c412c8 py3: make encodefun in store.py compatible with py3k
Mateusz Kwapich <mitrandir@fb.com>
parents: 30076
diff changeset
   171
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   172
    dmap = {}
48913
f254fc73d956 global: bulk replace simple pycompat.iteritems(x) with x.items()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
   173
    for k, v in cmap.items():
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   174
        dmap[v] = k
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   175
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   176
    def decode(s):
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   177
        i = 0
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   178
        while i < len(s):
49284
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
   179
            for l in range(1, 4):
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   180
                try:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   181
                    yield dmap[s[i : i + l]]
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   182
                    i += l
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   183
                    break
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   184
                except KeyError:
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   185
                    pass
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   186
            else:
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   187
                raise KeyError
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   188
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   189
    return (
49284
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
   190
        lambda s: b''.join([cmap[s[c : c + 1]] for c in range(len(s))]),
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   191
        lambda s: b''.join(list(decode(s))),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   192
    )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   193
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
   194
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
   195
_encodefname, _decodefname = _buildencodefun()
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   196
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   197
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
   198
def encodefilename(s):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   199
    """
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   200
    >>> encodefilename(b'foo.i/bar.d/bla.hg/hi:world?/HELLO')
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
   201
    'foo.i.hg/bar.d.hg/bla.hg.hg/hi~3aworld~3f/_h_e_l_l_o'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   202
    """
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
   203
    return _encodefname(encodedir(s))
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
   204
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   205
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
   206
def decodefilename(s):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   207
    """
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   208
    >>> decodefilename(b'foo.i.hg/bar.d.hg/bla.hg.hg/hi~3aworld~3f/_h_e_l_l_o')
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
   209
    'foo.i/bar.d/bla.hg/hi:world?/HELLO'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   210
    """
17608
776240123525 store: extract functions _encodefname and _decodefname
Adrian Buehlmann <adrian@cadifra.com>
parents: 17607
diff changeset
   211
    return decodedir(_decodefname(s))
6839
01db3e101362 move filename encoding functions from util.py to new store.py
Adrian Buehlmann <adrian@cadifra.com>
parents:
diff changeset
   212
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   213
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   214
def _buildlowerencodefun():
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   215
    """
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   216
    >>> f = _buildlowerencodefun()
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   217
    >>> f(b'nothing/special.txt')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   218
    'nothing/special.txt'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   219
    >>> f(b'HELLO')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   220
    'hello'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   221
    >>> f(b'hello:world?')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   222
    'hello~3aworld~3f'
34136
414a3513c2bd doctest: do not embed non-ascii characters in docstring
Yuya Nishihara <yuya@tcha.org>
parents: 34131
diff changeset
   223
    >>> f(b'the\\x07quick\\xADshot')
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   224
    'the~07quick~adshot'
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   225
    """
34214
7e3f078b6f31 py3: use bytechr() in store._buildlowerencodefun()
Yuya Nishihara <yuya@tcha.org>
parents: 34213
diff changeset
   226
    xchr = pycompat.bytechr
49284
d44e3c45f0e4 py3: replace `pycompat.xrange` by `range`
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
   227
    cmap = {xchr(x): xchr(x) for x in range(127)}
29071
2f58975eb2cb store: treat range as a generator instead of a list for py3 compat
timeless <timeless@mozdev.org>
parents: 28007
diff changeset
   228
    for x in _reserved():
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   229
        cmap[xchr(x)] = b"~%02x" % x
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   230
    for x in range(ord(b"A"), ord(b"Z") + 1):
34214
7e3f078b6f31 py3: use bytechr() in store._buildlowerencodefun()
Yuya Nishihara <yuya@tcha.org>
parents: 34213
diff changeset
   231
        cmap[xchr(x)] = xchr(x).lower()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   232
34213
96808804b68f store: give name to lowerencode function
Yuya Nishihara <yuya@tcha.org>
parents: 34136
diff changeset
   233
    def lowerencode(s):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   234
        return b"".join([cmap[c] for c in pycompat.iterbytestr(s)])
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   235
34213
96808804b68f store: give name to lowerencode function
Yuya Nishihara <yuya@tcha.org>
parents: 34136
diff changeset
   236
    return lowerencode
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   237
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   238
18430
0459c6555f69 store: implement lowerencode in C
Bryan O'Sullivan <bryano@fb.com>
parents: 18054
diff changeset
   239
lowerencode = getattr(parsers, 'lowerencode', None) or _buildlowerencodefun()
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   240
17570
f53a7b256ca6 store: optimze _auxencode() a bit by grouping the reserved names by length
Adrian Buehlmann <adrian@cadifra.com>
parents: 17569
diff changeset
   241
# Windows reserved names: con, prn, aux, nul, com1..com9, lpt1..lpt9
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   242
_winres3 = (b'aux', b'con', b'prn', b'nul')  # length 3
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   243
_winres4 = (b'com', b'lpt')  # length 4 (with trailing 1..9)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   244
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   245
12687
34d8247a4595 store: encode first period or space in filenames (issue1713)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12171
diff changeset
   246
def _auxencode(path, dotencode):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   247
    """
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   248
    Encodes filenames containing names reserved by Windows or which end in
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   249
    period or space. Does not touch other single reserved characters c.
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   250
    Specifically, c in '\\:*?"<>|' or ord(c) <= 31 are *not* encoded here.
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   251
    Additionally encodes space or period at the beginning, if dotencode is
17569
e9af2134825c store: explain "aux.foo" versus "foo.aux" in doc of _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17568
diff changeset
   252
    True. Parameter path is assumed to be all lowercase.
e9af2134825c store: explain "aux.foo" versus "foo.aux" in doc of _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17568
diff changeset
   253
    A segment only needs encoding if a reserved name appears as a
e9af2134825c store: explain "aux.foo" versus "foo.aux" in doc of _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17568
diff changeset
   254
    basename (e.g. "aux", "aux.foo"). A directory or file named "foo.aux"
e9af2134825c store: explain "aux.foo" versus "foo.aux" in doc of _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17568
diff changeset
   255
    doesn't need encoding.
13949
ba43aa1e173c store: add some doctests
Adrian Buehlmann <adrian@cadifra.com>
parents: 13426
diff changeset
   256
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   257
    >>> s = b'.foo/aux.txt/txt.aux/con/prn/nul/foo.'
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   258
    >>> _auxencode(s.split(b'/'), True)
17574
81a033bb29bc store: let _auxencode() return the list of path segments
Adrian Buehlmann <adrian@cadifra.com>
parents: 17573
diff changeset
   259
    ['~2efoo', 'au~78.txt', 'txt.aux', 'co~6e', 'pr~6e', 'nu~6c', 'foo~2e']
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   260
    >>> s = b'.com1com2/lpt9.lpt4.lpt1/conprn/com0/lpt0/foo.'
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   261
    >>> _auxencode(s.split(b'/'), False)
17574
81a033bb29bc store: let _auxencode() return the list of path segments
Adrian Buehlmann <adrian@cadifra.com>
parents: 17573
diff changeset
   262
    ['.com1com2', 'lp~749.lpt4.lpt1', 'conprn', 'com0', 'lpt0', 'foo~2e']
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   263
    >>> _auxencode([b'foo. '], True)
17574
81a033bb29bc store: let _auxencode() return the list of path segments
Adrian Buehlmann <adrian@cadifra.com>
parents: 17573
diff changeset
   264
    ['foo.~20']
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 33412
diff changeset
   265
    >>> _auxencode([b' .foo'], True)
17574
81a033bb29bc store: let _auxencode() return the list of path segments
Adrian Buehlmann <adrian@cadifra.com>
parents: 17573
diff changeset
   266
    ['~20.foo']
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   267
    """
17589
b11024849db6 store: parameter path of _auxencode is now a list of strings
Adrian Buehlmann <adrian@cadifra.com>
parents: 17588
diff changeset
   268
    for i, n in enumerate(path):
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
   269
        if not n:
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
   270
            continue
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   271
        if dotencode and n[0] in b'. ':
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   272
            n = b"~%02x" % ord(n[0:1]) + n[1:]
17589
b11024849db6 store: parameter path of _auxencode is now a list of strings
Adrian Buehlmann <adrian@cadifra.com>
parents: 17588
diff changeset
   273
            path[i] = n
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
   274
        else:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   275
            l = n.find(b'.')
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
   276
            if l == -1:
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
   277
                l = len(n)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   278
            if (l == 3 and n[:3] in _winres3) or (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   279
                l == 4
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   280
                and n[3:4] <= b'9'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   281
                and n[3:4] >= b'1'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   282
                and n[:3] in _winres4
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   283
            ):
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
   284
                # encode third letter ('aux' -> 'au~78')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   285
                ec = b"~%02x" % ord(n[2:3])
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
   286
                n = n[0:2] + ec + n[3:]
17589
b11024849db6 store: parameter path of _auxencode is now a list of strings
Adrian Buehlmann <adrian@cadifra.com>
parents: 17588
diff changeset
   287
                path[i] = n
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   288
        if n[-1] in b'. ':
17572
b644287e79a8 store: unindent most of the contents of the for loop in _auxencode()
Adrian Buehlmann <adrian@cadifra.com>
parents: 17571
diff changeset
   289
            # encode last period or space ('foo...' -> 'foo..~2e')
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   290
            path[i] = n[:-1] + b"~%02x" % ord(n[-1:])
17589
b11024849db6 store: parameter path of _auxencode is now a list of strings
Adrian Buehlmann <adrian@cadifra.com>
parents: 17588
diff changeset
   291
    return path
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   292
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   293
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   294
_maxstorepathlen = 120
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   295
_dirprefixlen = 8
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   296
_maxshortdirslen = 8 * (_dirprefixlen + 1) - 4
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   297
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   298
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   299
def _hashencode(path, dotencode):
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
   300
    digest = hex(hashutil.sha1(path).digest())
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   301
    le = lowerencode(path[5:]).split(b'/')  # skips prefix 'data/' or 'meta/'
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   302
    parts = _auxencode(le, dotencode)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   303
    basename = parts[-1]
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   304
    _root, ext = os.path.splitext(basename)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   305
    sdirs = []
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   306
    sdirslen = 0
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   307
    for p in parts[:-1]:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   308
        d = p[:_dirprefixlen]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   309
        if d[-1] in b'. ':
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   310
            # Windows can't access dirs ending in period or space
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   311
            d = d[:-1] + b'_'
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   312
        if sdirslen == 0:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   313
            t = len(d)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   314
        else:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   315
            t = sdirslen + 1 + len(d)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   316
            if t > _maxshortdirslen:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   317
                break
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   318
        sdirs.append(d)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   319
        sdirslen = t
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   320
    dirs = b'/'.join(sdirs)
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   321
    if len(dirs) > 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   322
        dirs += b'/'
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   323
    res = b'dh/' + dirs + digest + ext
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   324
    spaceleft = _maxstorepathlen - len(res)
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   325
    if spaceleft > 0:
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   326
        filler = basename[:spaceleft]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   327
        res = b'dh/' + dirs + filler + digest + ext
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   328
    return res
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   329
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   330
17590
eb0884680f5c store: eliminate one level of lambda functions on _hybridencode
Adrian Buehlmann <adrian@cadifra.com>
parents: 17589
diff changeset
   331
def _hybridencode(path, dotencode):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   332
    """encodes path with a length limit
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   333
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   334
    Encodes all paths that begin with 'data/', according to the following.
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   335
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   336
    Default encoding (reversible):
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   337
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   338
    Encodes all uppercase letters 'X' as '_x'. All reserved or illegal
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   339
    characters are encoded as '~xx', where xx is the two digit hex code
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   340
    of the character (see encodefilename).
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   341
    Relevant path components consisting of Windows reserved filenames are
17738
b8424c92ba2b spelling: fix minor spell checker issues
Mads Kiilerich <mads@kiilerich.com>
parents: 17731
diff changeset
   342
    masked by encoding the third character ('aux' -> 'au~78', see _auxencode).
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   343
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   344
    Hashed encoding (not reversible):
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   345
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   346
    If the default-encoded path is longer than _maxstorepathlen, a
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   347
    non-reversible hybrid hashing of the path is done instead.
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   348
    This encoding uses up to _dirprefixlen characters of all directory
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   349
    levels of the lowerencoded path, but not more levels than can fit into
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   350
    _maxshortdirslen.
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   351
    Then follows the filler followed by the sha digest of the full path.
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   352
    The filler is the beginning of the basename of the lowerencoded path
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   353
    (the basename is everything after the last path separator). The filler
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   354
    is as long as possible, filling in characters from the basename until
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   355
    the encoded path has _maxstorepathlen characters (or all chars of the
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   356
    basename have been taken).
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   357
    The extension (e.g. '.i' or '.d') is preserved.
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   358
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   359
    The string 'data/' at the beginning is replaced with 'dh/', if the hashed
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   360
    encoding was used.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   361
    """
17609
cbc180cfd60b store: reuse direncoded path in _hybridencode
Adrian Buehlmann <adrian@cadifra.com>
parents: 17608
diff changeset
   362
    path = encodedir(path)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   363
    ef = _encodefname(path).split(b'/')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   364
    res = b'/'.join(_auxencode(ef, dotencode))
14288
00a0ab08f986 store: change names to comply with project coding standards
Adrian Buehlmann <adrian@cadifra.com>
parents: 14194
diff changeset
   365
    if len(res) > _maxstorepathlen:
17610
d0afa149e059 store: refactor hashed encoding into its own function
Bryan O'Sullivan <bryano@fb.com>
parents: 17609
diff changeset
   366
        res = _hashencode(path, dotencode)
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   367
    return res
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
   368
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   369
17624
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
   370
def _pathencode(path):
18435
8c019d2fd7c0 store: switch to C-based hashed path encoding
Bryan O'Sullivan <bryano@fb.com>
parents: 18430
diff changeset
   371
    de = encodedir(path)
17693
0c6de45e1212 store: optimize _pathencode by checking the length of the unencoded path
Adrian Buehlmann <adrian@cadifra.com>
parents: 17653
diff changeset
   372
    if len(path) > _maxstorepathlen:
18435
8c019d2fd7c0 store: switch to C-based hashed path encoding
Bryan O'Sullivan <bryano@fb.com>
parents: 18430
diff changeset
   373
        return _hashencode(de, True)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   374
    ef = _encodefname(de).split(b'/')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   375
    res = b'/'.join(_auxencode(ef, True))
17624
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
   376
    if len(res) > _maxstorepathlen:
18435
8c019d2fd7c0 store: switch to C-based hashed path encoding
Bryan O'Sullivan <bryano@fb.com>
parents: 18430
diff changeset
   377
        return _hashencode(de, True)
17624
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
   378
    return res
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
   379
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   380
17624
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
   381
_pathencode = getattr(parsers, 'pathencode', _pathencode)
ae103510f6aa store: add a fallback _pathencode Python function
Adrian Buehlmann <adrian@cadifra.com>
parents: 17623
diff changeset
   382
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   383
17623
448e6ed7c557 store: move _plainhybridencode and _dothybridencode higher up in the file
Adrian Buehlmann <adrian@cadifra.com>
parents: 17621
diff changeset
   384
def _plainhybridencode(f):
448e6ed7c557 store: move _plainhybridencode and _dothybridencode higher up in the file
Adrian Buehlmann <adrian@cadifra.com>
parents: 17621
diff changeset
   385
    return _hybridencode(f, False)
448e6ed7c557 store: move _plainhybridencode and _dothybridencode higher up in the file
Adrian Buehlmann <adrian@cadifra.com>
parents: 17621
diff changeset
   386
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   387
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
   388
def _calcmode(vfs):
6898
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
   389
    try:
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
   390
        # files in .hg/ will be created using this mode
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
   391
        mode = vfs.stat().st_mode
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   392
        # avoid some useless chmods
25658
e93036747902 global: mass rewrite to use modern octal syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25091
diff changeset
   393
        if (0o777 & ~util.umask) == (0o777 & mode):
6898
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
   394
            mode = None
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
   395
    except OSError:
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
   396
        mode = None
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
   397
    return mode
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
   398
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   399
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   400
_data = [
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   401
    b'bookmarks',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   402
    b'narrowspec',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   403
    b'data',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   404
    b'meta',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   405
    b'00manifest.d',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   406
    b'00manifest.i',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   407
    b'00changelog.d',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   408
    b'00changelog.i',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   409
    b'phaseroots',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   410
    b'obsstore',
45483
d252f51ab032 share: introduce config option to store requires in .hg/store
Pulkit Goyal <7895pulkit@gmail.com>
parents: 45351
diff changeset
   411
    b'requires',
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   412
]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   413
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   414
REVLOG_FILES_EXT = (
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   415
    b'.i',
47324
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
   416
    b'.idx',
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
   417
    b'.d',
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
   418
    b'.dat',
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
   419
    b'.n',
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
   420
    b'.nd',
47389
e6292eb33384 revlog: store sidedata in their own file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47324
diff changeset
   421
    b'.sda',
47324
0a3fa41fa719 revlogv2: use a unique filename for data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47234
diff changeset
   422
)
50476
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   423
# file extension that also use a `-SOMELONGIDHASH.ext` form
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   424
REVLOG_FILES_LONG_EXT = (
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   425
    b'.nd',
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   426
    b'.idx',
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   427
    b'.dat',
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   428
    b'.sda',
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   429
)
46989
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
   430
# files that are "volatile" and might change between listing and streaming
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
   431
#
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
   432
# note: the ".nd" file are nodemap data and won't "change" but they might be
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
   433
# deleted.
aed6ceaad6d7 streamclone: treat volatile file as "fullfile"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46895
diff changeset
   434
REVLOG_FILES_VOLATILE_EXT = (b'.n', b'.nd')
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
   435
46990
0b569c75d180 store: exclude `undo.` nodemap's file from `walk`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
   436
# some exception to the above matching
47659
f030c7d22032 walk: no longer ignore revlogs of files starting with `undo.` (issue6542)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47443
diff changeset
   437
#
f030c7d22032 walk: no longer ignore revlogs of files starting with `undo.` (issue6542)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47443
diff changeset
   438
# XXX This is currently not in use because of issue6542
50319
4be9ecc982e1 py3: fix for Python 3.12 emitting SyntaxWarning on invalid escape sequences
Mads Kiilerich <mads@kiilerich.com>
parents: 50315
diff changeset
   439
EXCLUDED = re.compile(br'.*undo\.[^/]+\.(nd?|i)$')
46990
0b569c75d180 store: exclude `undo.` nodemap's file from `walk`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46989
diff changeset
   440
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
   441
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
   442
def is_revlog(f, kind, st):
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
   443
    if kind != stat.S_IFREG:
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   444
        return False
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   445
    if f.endswith(REVLOG_FILES_EXT):
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   446
        return True
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   447
    return False
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
   448
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
   449
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   450
def is_revlog_file(f):
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   451
    if f.endswith(REVLOG_FILES_EXT):
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   452
        return True
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   453
    return False
46383
374d7fff7cb5 store: use `endswith` to detect revlog extension
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46322
diff changeset
   454
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
   455
50635
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   456
@attr.s(slots=True)
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   457
class StoreFile:
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   458
    """a file matching a store entry"""
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   459
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   460
    unencoded_path = attr.ib()
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   461
    _file_size = attr.ib(default=None)
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   462
    is_volatile = attr.ib(default=False)
52359
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
   463
    # Missing file can be safely ignored, used by "copy/hardlink" local clone
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
   464
    # for cache file not covered by lock.
3f0cf7bb3086 stream: preserve volatile cache early
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52357
diff changeset
   465
    optional = False
50635
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   466
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   467
    def file_size(self, vfs):
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   468
        if self._file_size is None:
50636
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   469
            if vfs is None:
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   470
                msg = b"calling vfs-less file_size without prior call: %s"
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   471
                msg %= self.unencoded_path
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   472
                raise error.ProgrammingError(msg)
50635
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   473
            try:
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   474
                self._file_size = vfs.stat(self.unencoded_path).st_size
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   475
            except FileNotFoundError:
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   476
                self._file_size = 0
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   477
        return self._file_size
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   478
51544
8e2a68e10f5c stream: in v3, skip the "size" fast path if the entries as some unknown size
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50926
diff changeset
   479
    @property
8e2a68e10f5c stream: in v3, skip the "size" fast path if the entries as some unknown size
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50926
diff changeset
   480
    def has_size(self):
8e2a68e10f5c stream: in v3, skip the "size" fast path if the entries as some unknown size
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50926
diff changeset
   481
        return self._file_size is not None
8e2a68e10f5c stream: in v3, skip the "size" fast path if the entries as some unknown size
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50926
diff changeset
   482
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51878
diff changeset
   483
    def get_stream(self, vfs, volatiles):
50636
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   484
        """return data "stream" information for this file
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   485
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   486
        (unencoded_file_path, content_iterator, content_size)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   487
        """
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   488
        size = self.file_size(None)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   489
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   490
        def get_stream():
52013
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
   491
            path = vfs.join(self.unencoded_path)
0ad269e24075 stream: open volatile file through the manager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52012
diff changeset
   492
            with volatiles.open(path) as fp:
50636
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   493
                yield None  # ready to stream
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   494
                if size <= 65536:
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   495
                    yield fp.read(size)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   496
                else:
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   497
                    yield from util.filechunkiter(fp, limit=size)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   498
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   499
        s = get_stream()
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   500
        next(s)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   501
        return (self.unencoded_path, s, size)
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   502
50635
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   503
50471
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   504
@attr.s(slots=True, init=False)
50470
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   505
class BaseStoreEntry:
50468
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50466
diff changeset
   506
    """An entry in the store
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50466
diff changeset
   507
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50466
diff changeset
   508
    This is returned by `store.walk` and represent some data in the store."""
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50466
diff changeset
   509
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   510
    maybe_volatile = True
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   511
50635
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   512
    def files(self) -> List[StoreFile]:
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   513
        raise NotImplementedError
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   514
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   515
    def get_streams(
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   516
        self,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   517
        repo=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   518
        vfs=None,
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51878
diff changeset
   519
        volatiles=None,
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   520
        max_changeset=None,
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   521
        preserve_file_count=False,
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   522
    ):
50636
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   523
        """return a list of data stream associated to files for this entry
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   524
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   525
        return [(unencoded_file_path, content_iterator, content_size), …]
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   526
        """
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   527
        assert vfs is not None
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51878
diff changeset
   528
        return [f.get_stream(vfs, volatiles) for f in self.files()]
50636
5e60abf811f3 stream-clone: make it the responsability of the store entry to stream content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50635
diff changeset
   529
52357
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   530
    def preserve_volatiles(self, vfs, volatiles):
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   531
        """Use a VolatileManager to preserve the state of any volatile file
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   532
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   533
        This is useful for code that need a consistent view of the content like stream clone.
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   534
        """
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   535
        if self.maybe_volatile:
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   536
            for f in self.files():
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   537
                if f.is_volatile:
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   538
                    volatiles(vfs.join(f.unencoded_path))
a260d326458f stream: create a `e.preserve_volatiles` method directly on StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 52171
diff changeset
   539
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   540
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   541
@attr.s(slots=True, init=False)
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   542
class SimpleStoreEntry(BaseStoreEntry):
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   543
    """A generic entry in the store"""
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   544
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   545
    is_revlog = False
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   546
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   547
    maybe_volatile = attr.ib()
50492
ed8cda1c18e1 store: rename `unencoded_path` to `entry_path` for StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50491
diff changeset
   548
    _entry_path = attr.ib()
50473
2b2284cf949b store: only access is_volatile information through the file object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
   549
    _is_volatile = attr.ib(default=False)
50472
7d4d2a160cf5 store: only access file_size information through the file object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50471
diff changeset
   550
    _file_size = attr.ib(default=None)
50628
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   551
    _files = attr.ib(default=None)
50468
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50466
diff changeset
   552
50471
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   553
    def __init__(
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   554
        self,
50492
ed8cda1c18e1 store: rename `unencoded_path` to `entry_path` for StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50491
diff changeset
   555
        entry_path,
50471
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   556
        is_volatile=False,
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   557
        file_size=None,
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   558
    ):
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   559
        super().__init__()
50492
ed8cda1c18e1 store: rename `unencoded_path` to `entry_path` for StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50491
diff changeset
   560
        self._entry_path = entry_path
50473
2b2284cf949b store: only access is_volatile information through the file object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50472
diff changeset
   561
        self._is_volatile = is_volatile
50472
7d4d2a160cf5 store: only access file_size information through the file object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50471
diff changeset
   562
        self._file_size = file_size
50628
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   563
        self._files = None
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   564
        self.maybe_volatile = is_volatile
50471
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   565
50635
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   566
    def files(self) -> List[StoreFile]:
50628
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   567
        if self._files is None:
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   568
            self._files = [
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   569
                StoreFile(
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   570
                    unencoded_path=self._entry_path,
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   571
                    file_size=self._file_size,
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   572
                    is_volatile=self._is_volatile,
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   573
                )
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   574
            ]
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   575
        return self._files
50469
9fdc28e21b68 store: introduce a EntryFile object to actually access file info
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50468
diff changeset
   576
9fdc28e21b68 store: introduce a EntryFile object to actually access file info
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50468
diff changeset
   577
50471
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   578
@attr.s(slots=True, init=False)
50470
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   579
class RevlogStoreEntry(BaseStoreEntry):
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   580
    """A revlog entry in the store"""
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   581
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   582
    is_revlog = True
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   583
50470
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   584
    revlog_type = attr.ib(default=None)
50480
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
   585
    target_id = attr.ib(default=None)
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   586
    maybe_volatile = attr.ib(default=True)
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   587
    _path_prefix = attr.ib(default=None)
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   588
    _details = attr.ib(default=None)
50628
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   589
    _files = attr.ib(default=None)
50470
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   590
50471
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   591
    def __init__(
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   592
        self,
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   593
        revlog_type,
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   594
        path_prefix,
50480
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
   595
        target_id,
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   596
        details,
50471
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   597
    ):
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   598
        super().__init__()
50471
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   599
        self.revlog_type = revlog_type
50480
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
   600
        self.target_id = target_id
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   601
        self._path_prefix = path_prefix
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   602
        assert b'.i' in details, (path_prefix, details)
51549
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   603
        for ext in details:
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   604
            if ext.endswith(REVLOG_FILES_VOLATILE_EXT):
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   605
                self.maybe_volatile = True
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   606
                break
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   607
        else:
49faa72b994e streamclone: stop listing files for entries that have no volatile files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51544
diff changeset
   608
            self.maybe_volatile = False
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   609
        self._details = details
50628
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   610
        self._files = None
50471
c37450a5f1dc store: have custom init for entries class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50470
diff changeset
   611
50494
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
   612
    @property
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
   613
    def is_changelog(self):
50677
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
   614
        return self.revlog_type == KIND_CHANGELOG
50494
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
   615
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
   616
    @property
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
   617
    def is_manifestlog(self):
50677
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
   618
        return self.revlog_type == KIND_MANIFESTLOG
50494
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
   619
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
   620
    @property
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
   621
    def is_filelog(self):
50677
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
   622
        return self.revlog_type == KIND_FILELOG
50494
66c556968222 store: introduce boolean property for revlog type
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50493
diff changeset
   623
50489
3473d18c029a store: introduce a main_file_path method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
   624
    def main_file_path(self):
3473d18c029a store: introduce a main_file_path method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
   625
        """unencoded path of the main revlog file"""
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   626
        return self._path_prefix + b'.i'
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   627
50635
8fc10bfd9887 store: declare a `files` method on BaseStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50629
diff changeset
   628
    def files(self) -> List[StoreFile]:
50628
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   629
        if self._files is None:
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   630
            self._files = []
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   631
            for ext in sorted(self._details, key=_ext_key):
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   632
                path = self._path_prefix + ext
50667
11562d72cb7b store: directly pass the filesize in the `details` of revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50666
diff changeset
   633
                file_size = self._details[ext]
50665
6b522a9e7451 store: have the revlog determine which files are volatile itself
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
   634
                # files that are "volatile" and might change between
6b522a9e7451 store: have the revlog determine which files are volatile itself
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
   635
                # listing and streaming
6b522a9e7451 store: have the revlog determine which files are volatile itself
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
   636
                #
6b522a9e7451 store: have the revlog determine which files are volatile itself
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
   637
                # note: the ".nd" file are nodemap data and won't "change"
6b522a9e7451 store: have the revlog determine which files are volatile itself
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
   638
                # but they might be deleted.
6b522a9e7451 store: have the revlog determine which files are volatile itself
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
   639
                volatile = ext.endswith(REVLOG_FILES_VOLATILE_EXT)
50666
cdb471c8ebcf store: explicitly pass file_size when creating StoreFile
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50665
diff changeset
   640
                f = StoreFile(path, file_size, volatile)
50665
6b522a9e7451 store: have the revlog determine which files are volatile itself
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50638
diff changeset
   641
                self._files.append(f)
50628
b59e0a4f692f store: cache the `files()` return for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50626
diff changeset
   642
        return self._files
50489
3473d18c029a store: introduce a main_file_path method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50480
diff changeset
   643
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   644
    def get_streams(
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   645
        self,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   646
        repo=None,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   647
        vfs=None,
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51878
diff changeset
   648
        volatiles=None,
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   649
        max_changeset=None,
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   650
        preserve_file_count=False,
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   651
    ):
51544
8e2a68e10f5c stream: in v3, skip the "size" fast path if the entries as some unknown size
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50926
diff changeset
   652
        pre_sized = all(f.has_size for f in self.files())
8e2a68e10f5c stream: in v3, skip the "size" fast path if the entries as some unknown size
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50926
diff changeset
   653
        if pre_sized and (
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   654
            repo is None
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   655
            or max_changeset is None
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   656
            # This use revlog-v2, ignore for now
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   657
            or any(k.endswith(b'.idx') for k in self._details.keys())
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   658
            # This is not inline, no race expected
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   659
            or b'.d' in self._details
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   660
        ):
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   661
            return super().get_streams(
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   662
                repo=repo,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   663
                vfs=vfs,
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51878
diff changeset
   664
                volatiles=volatiles,
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   665
                max_changeset=max_changeset,
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   666
                preserve_file_count=preserve_file_count,
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   667
            )
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   668
        elif not preserve_file_count:
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   669
            stream = [
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51878
diff changeset
   670
                f.get_stream(vfs, volatiles)
50703
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   671
                for f in self.files()
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   672
                if not f.unencoded_path.endswith((b'.i', b'.d'))
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   673
            ]
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   674
            rl = self.get_revlog_instance(repo).get_revlog()
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   675
            rl_stream = rl.get_streams(max_changeset)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   676
            stream.extend(rl_stream)
0452af304808 stream-clone: add a v3 version of the protocol
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50677
diff changeset
   677
            return stream
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   678
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   679
        name_to_size = {}
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   680
        for f in self.files():
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   681
            name_to_size[f.unencoded_path] = f.file_size(None)
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   682
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   683
        stream = [
52012
e308439339e2 stream: rename TempCopyManager to VolatileManager
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51878
diff changeset
   684
            f.get_stream(vfs, volatiles)
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   685
            for f in self.files()
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   686
            if not f.unencoded_path.endswith(b'.i')
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   687
        ]
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   688
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   689
        index_path = self._path_prefix + b'.i'
50638
5460424092e2 stream-clone: smoothly detect and handle a case were a revlog is split
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50637
diff changeset
   690
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   691
        index_file = None
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   692
        try:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   693
            index_file = vfs(index_path)
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   694
            header = index_file.read(INDEX_HEADER.size)
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   695
            if revlogmod.revlog.is_inline_index(header):
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   696
                size = name_to_size[index_path]
50638
5460424092e2 stream-clone: smoothly detect and handle a case were a revlog is split
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50637
diff changeset
   697
50669
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   698
                # no split underneath, just return the stream
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   699
                def get_stream():
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   700
                    fp = index_file
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   701
                    try:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   702
                        fp.seek(0)
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   703
                        yield None
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   704
                        if size <= 65536:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   705
                            yield fp.read(size)
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   706
                        else:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   707
                            yield from util.filechunkiter(fp, limit=size)
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   708
                    finally:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   709
                        fp.close()
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   710
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   711
                s = get_stream()
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   712
                next(s)
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   713
                index_file = None
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   714
                stream.append((index_path, s, size))
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   715
            else:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   716
                rl = self.get_revlog_instance(repo).get_revlog()
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   717
                rl_stream = rl.get_streams(max_changeset, force_inline=True)
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   718
                for name, s, size in rl_stream:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   719
                    if name_to_size.get(name, 0) != size:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   720
                        msg = _(b"expected %d bytes but %d provided for %s")
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   721
                        msg %= name_to_size.get(name, 0), size, name
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   722
                        raise error.Abort(msg)
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   723
                stream.extend(rl_stream)
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   724
        finally:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   725
            if index_file is not None:
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   726
                index_file.close()
3b56395404a1 stream-clone: avoid opening a revlog in case we do not need it
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50668
diff changeset
   727
50637
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   728
        files = self.files()
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   729
        assert len(stream) == len(files), (
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   730
            stream,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   731
            files,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   732
            self._path_prefix,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   733
            self.target_id,
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   734
        )
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   735
        return stream
9caa860dcbec stream-clone: implement decidated `get_streams` method for revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50636
diff changeset
   736
50624
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   737
    def get_revlog_instance(self, repo):
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   738
        """Obtain a revlog instance from this store entry
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   739
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   740
        An instance of the appropriate class is returned.
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   741
        """
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   742
        if self.is_changelog:
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   743
            return changelog.changelog(repo.svfs)
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   744
        elif self.is_manifestlog:
50625
3ea3767c23a4 store: do not drop the final `/` when creating manifestlog instance
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50624
diff changeset
   745
            mandir = self.target_id
50624
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   746
            return manifest.manifestrevlog(
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   747
                repo.nodeconstants, repo.svfs, tree=mandir
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   748
            )
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   749
        else:
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   750
            return filelog.filelog(repo.svfs, self.target_id)
e1ee6910f6bc store: add a `get_revlog_instance` method on revlog entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50511
diff changeset
   751
50470
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   752
50476
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   753
def _gather_revlog(files_data):
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   754
    """group files per revlog prefix
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   755
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   756
    The returns a two level nested dict. The top level key is the revlog prefix
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   757
    without extension, the second level is all the file "suffix" that were
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   758
    seen for this revlog and arbitrary file data as value.
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   759
    """
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   760
    revlogs = collections.defaultdict(dict)
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   761
    for u, value in files_data:
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   762
        name, ext = _split_revlog_ext(u)
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   763
        revlogs[name][ext] = value
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   764
    return sorted(revlogs.items())
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   765
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   766
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   767
def _split_revlog_ext(filename):
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   768
    """split the revlog file prefix from the variable extension"""
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   769
    if filename.endswith(REVLOG_FILES_LONG_EXT):
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   770
        char = b'-'
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   771
    else:
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   772
        char = b'.'
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   773
    idx = filename.rfind(char)
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   774
    return filename[:idx], filename[idx:]
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   775
5217e36356bb store: add logic to group revlog file together
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50475
diff changeset
   776
50478
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   777
def _ext_key(ext):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   778
    """a key to order revlog suffix
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   779
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   780
    important to issue .i after other entry."""
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   781
    # the only important part of this order is to keep the `.i` last.
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   782
    if ext.endswith(b'.n'):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   783
        return (0, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   784
    elif ext.endswith(b'.nd'):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   785
        return (10, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   786
    elif ext.endswith(b'.d'):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   787
        return (20, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   788
    elif ext.endswith(b'.i'):
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   789
        return (50, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   790
    else:
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   791
        return (40, ext)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   792
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   793
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48941
diff changeset
   794
class basicstore:
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   795
    '''base class for local repository stores'''
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   796
17651
3b49c28658f6 store: rename "openertype" argument to "vfstype"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17649
diff changeset
   797
    def __init__(self, path, vfstype):
17724
bf4b72d8dd4d store: initialize vfs field first to use it for initialization of others
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17722
diff changeset
   798
        vfs = vfstype(path)
bf4b72d8dd4d store: initialize vfs field first to use it for initialization of others
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17722
diff changeset
   799
        self.path = vfs.base
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
   800
        self.createmode = _calcmode(vfs)
17652
2c6f7231becc store: rename "op" variables to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17651
diff changeset
   801
        vfs.createmode = self.createmode
17728
004bd533880d store: invoke "os.path.isdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17727
diff changeset
   802
        self.rawvfs = vfs
31234
9b7a2ef4f27c vfs: use 'vfs' module directly in 'mercurial.store'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31219
diff changeset
   803
        self.vfs = vfsmod.filtervfs(vfs, encodedir)
17653
dacb50696b75 store: initialize "vfs" fields by "vfs" constructors
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17652
diff changeset
   804
        self.opener = self.vfs
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   805
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   806
    def join(self, f):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   807
        return self.path + b'/' + encodedir(f)
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   808
50490
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   809
    def _walk(self, relpath, recurse, undecodable=None):
47877
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   810
        '''yields (revlog_type, unencoded, size)'''
13426
643b8212813e store: remove pointless pathjoiner parameter
Adrian Buehlmann <adrian@cadifra.com>
parents: 13391
diff changeset
   811
        path = self.path
643b8212813e store: remove pointless pathjoiner parameter
Adrian Buehlmann <adrian@cadifra.com>
parents: 13391
diff changeset
   812
        if relpath:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   813
            path += b'/' + relpath
13426
643b8212813e store: remove pointless pathjoiner parameter
Adrian Buehlmann <adrian@cadifra.com>
parents: 13391
diff changeset
   814
        striplen = len(self.path) + 1
6899
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
   815
        l = []
17728
004bd533880d store: invoke "os.path.isdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17727
diff changeset
   816
        if self.rawvfs.isdir(path):
6899
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
   817
            visit = [path]
17747
aad3bce98f76 store: invoke "osutil.listdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17745
diff changeset
   818
            readdir = self.rawvfs.readdir
6899
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
   819
            while visit:
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
   820
                p = visit.pop()
17747
aad3bce98f76 store: invoke "osutil.listdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17745
diff changeset
   821
                for f, kind, st in readdir(p, stat=True):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   822
                    fp = p + b'/' + f
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   823
                    if is_revlog(f, kind, st):
6900
def492d1b592 store: change handling of decoding errors
Matt Mackall <mpm@selenic.com>
parents: 6899
diff changeset
   824
                        n = util.pconvert(fp[striplen:])
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   825
                        l.append((decodedir(n), st.st_size))
6899
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
   826
                    elif kind == stat.S_IFDIR and recurse:
56a7a54e074f store: simplify walking
Matt Mackall <mpm@selenic.com>
parents: 6898
diff changeset
   827
                        visit.append(fp)
50475
1c0244a8cdaf store: change `_walk` return to `(filename, (type, size))`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50474
diff changeset
   828
17054
125ff5654b72 store: sort filenames in place
Bryan O'Sullivan <bryano@fb.com>
parents: 16404
diff changeset
   829
        l.sort()
125ff5654b72 store: sort filenames in place
Bryan O'Sullivan <bryano@fb.com>
parents: 16404
diff changeset
   830
        return l
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   831
46607
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
   832
    def changelog(self, trypending, concurrencychecker=None):
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
   833
        return changelog.changelog(
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
   834
            self.vfs,
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
   835
            trypending=trypending,
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
   836
            concurrencychecker=concurrencychecker,
e9901d01d135 revlog: add a mechanism to verify expected file position before appending
Kyle Lippincott <spectral@google.com>
parents: 46383
diff changeset
   837
        )
42905
3df3b139a43d localrepo: push manifestlog and changelog construction code into store
Augie Fackler <augie@google.com>
parents: 42715
diff changeset
   838
52066
6641a3ae8313 manifest: drop the CamelCase name for `manifest.manifestlog`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52013
diff changeset
   839
    def manifestlog(self, repo, storenarrowmatch) -> manifest.manifestlog:
46780
6266d19556ad node: introduce nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents: 46607
diff changeset
   840
        rootstore = manifest.manifestrevlog(repo.nodeconstants, self.vfs)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   841
        return manifest.manifestlog(self.vfs, repo, rootstore, storenarrowmatch)
42905
3df3b139a43d localrepo: push manifestlog and changelog construction code into store
Augie Fackler <augie@google.com>
parents: 42715
diff changeset
   842
50501
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
   843
    def data_entries(
50468
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50466
diff changeset
   844
        self, matcher=None, undecodable=None
50470
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   845
    ) -> Generator[BaseStoreEntry, None, None]:
47877
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   846
        """Like walk, but excluding the changelog and root manifest.
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   847
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   848
        When [undecodable] is None, revlogs names that can't be
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   849
        decoded cause an exception. When it is provided, it should
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   850
        be a list and the filenames that can't be decoded are added
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   851
        to it instead. This is very rarely needed."""
50479
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
   852
        dirs = [
50677
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
   853
            (b'data', KIND_FILELOG, False),
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
   854
            (b'meta', KIND_MANIFESTLOG, True),
50479
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
   855
        ]
50626
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50625
diff changeset
   856
        for base_dir, rl_type, strip_filename in dirs:
50490
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   857
            files = self._walk(base_dir, True, undecodable=undecodable)
50479
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
   858
            for revlog, details in _gather_revlog(files):
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   859
                revlog_target_id = revlog.split(b'/', 1)[1]
50626
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50625
diff changeset
   860
                if strip_filename and b'/' in revlog:
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50625
diff changeset
   861
                    revlog_target_id = revlog_target_id.rsplit(b'/', 1)[0]
309cbd8400ae store: properly compute the targer_id of manifestlog in no-fncache walk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50625
diff changeset
   862
                    revlog_target_id += b'/'
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   863
                yield RevlogStoreEntry(
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   864
                    path_prefix=revlog,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   865
                    revlog_type=rl_type,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   866
                    target_id=revlog_target_id,
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   867
                    details=details,
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   868
                )
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   869
50511
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   870
    def top_entries(
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   871
        self, phase=False, obsolescence=False
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   872
    ) -> Generator[BaseStoreEntry, None, None]:
50510
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50509
diff changeset
   873
        if phase and self.vfs.exists(b'phaseroots'):
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50509
diff changeset
   874
            yield SimpleStoreEntry(
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50509
diff changeset
   875
                entry_path=b'phaseroots',
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50509
diff changeset
   876
                is_volatile=True,
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50509
diff changeset
   877
            )
5a62d56e3955 store: yield phases before changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50509
diff changeset
   878
50511
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   879
        if obsolescence and self.vfs.exists(b'obsstore'):
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   880
            # XXX if we had the file size it could be non-volatile
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   881
            yield SimpleStoreEntry(
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   882
                entry_path=b'obsstore',
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   883
                is_volatile=True,
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   884
            )
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   885
50478
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   886
        files = reversed(self._walk(b'', False))
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   887
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   888
        changelogs = collections.defaultdict(dict)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   889
        manifestlogs = collections.defaultdict(dict)
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   890
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   891
        for u, s in files:
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
   892
            if u.startswith(b'00changelog'):
50478
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   893
                name, ext = _split_revlog_ext(u)
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   894
                changelogs[name][ext] = s
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
   895
            elif u.startswith(b'00manifest'):
50478
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   896
                name, ext = _split_revlog_ext(u)
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   897
                manifestlogs[name][ext] = s
46895
6085b7f1536d store: also return some information about the type of file `walk` found
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46854
diff changeset
   898
            else:
50470
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   899
                yield SimpleStoreEntry(
50492
ed8cda1c18e1 store: rename `unencoded_path` to `entry_path` for StoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50491
diff changeset
   900
                    entry_path=u,
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   901
                    is_volatile=False,
50470
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   902
                    file_size=s,
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
   903
                )
50478
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   904
        # yield manifest before changelog
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   905
        top_rl = [
50677
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
   906
            (manifestlogs, KIND_MANIFESTLOG),
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
   907
            (changelogs, KIND_CHANGELOG),
50478
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   908
        ]
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   909
        assert len(manifestlogs) <= 1
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   910
        assert len(changelogs) <= 1
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   911
        for data, revlog_type in top_rl:
b08243dbc2e3 store: also gather files per revlog in `topfiles`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50477
diff changeset
   912
            for revlog, details in sorted(data.items()):
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   913
                yield RevlogStoreEntry(
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   914
                    path_prefix=revlog,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   915
                    revlog_type=revlog_type,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   916
                    target_id=b'',
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
   917
                    details=details,
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
   918
                )
19177
1e104aaa4c44 store: move top file walk to a separate function
Durham Goode <durham@fb.com>
parents: 18435
diff changeset
   919
50509
a32d739b0ffb store: make `walk` return an entry for phase if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50502
diff changeset
   920
    def walk(
50511
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   921
        self, matcher=None, phase=False, obsolescence=False
50509
a32d739b0ffb store: make `walk` return an entry for phase if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50502
diff changeset
   922
    ) -> Generator[BaseStoreEntry, None, None]:
50448
584ff1f97201 comments: fix spelling
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50319
diff changeset
   923
        """return files related to data storage (ie: revlogs)
46853
eed3e2b79b48 store: document the `walk` method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46819
diff changeset
   924
50509
a32d739b0ffb store: make `walk` return an entry for phase if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50502
diff changeset
   925
        yields instance from BaseStoreEntry subclasses
40340
2d45b549392f store: pass matcher to store.datafiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39698
diff changeset
   926
2d45b549392f store: pass matcher to store.datafiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39698
diff changeset
   927
        if a matcher is passed, storage files of only those tracked paths
2d45b549392f store: pass matcher to store.datafiles()
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 39698
diff changeset
   928
        are passed with matches the matcher
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
   929
        """
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   930
        # yield data files first
50501
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
   931
        for x in self.data_entries(matcher):
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   932
            yield x
50511
0925eaf09c8b store: make `walk` return an entry for obsolescence if requested so
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   933
        for x in self.top_entries(phase=phase, obsolescence=obsolescence):
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   934
            yield x
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   935
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
   936
    def copylist(self):
45483
d252f51ab032 share: introduce config option to store requires in .hg/store
Pulkit Goyal <7895pulkit@gmail.com>
parents: 45351
diff changeset
   937
        return _data
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
   938
20883
cd443c7589cc fncache: move fncache writing to be in a transaction
Durham Goode <durham@fb.com>
parents: 20879
diff changeset
   939
    def write(self, tr):
13391
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
   940
        pass
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
   941
20884
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
   942
    def invalidatecaches(self):
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
   943
        pass
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
   944
20885
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
   945
    def markremoved(self, fn):
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
   946
        pass
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
   947
17744
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
   948
    def __contains__(self, path):
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
   949
        '''Checks if the store contains path'''
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   950
        path = b"/".join((b"data", path))
17744
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
   951
        # file?
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   952
        if self.vfs.exists(path + b".i"):
17744
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
   953
            return True
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
   954
        # dir?
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   955
        if not path.endswith(b"/"):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   956
            path = path + b"/"
19903
ca875b271ac3 store: use "vfs.exists()" instead of "os.path.exists()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 19177
diff changeset
   957
        return self.vfs.exists(path)
17744
09d5b2055295 store: add a contains method to basicstore
smuralid
parents: 17738
diff changeset
   958
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
   959
6898
69aeaaaf6e07 store: simplify class hierarchy
Matt Mackall <mpm@selenic.com>
parents: 6897
diff changeset
   960
class encodedstore(basicstore):
17651
3b49c28658f6 store: rename "openertype" argument to "vfstype"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17649
diff changeset
   961
    def __init__(self, path, vfstype):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   962
        vfs = vfstype(path + b'/store')
17724
bf4b72d8dd4d store: initialize vfs field first to use it for initialization of others
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17722
diff changeset
   963
        self.path = vfs.base
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
   964
        self.createmode = _calcmode(vfs)
17652
2c6f7231becc store: rename "op" variables to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17651
diff changeset
   965
        vfs.createmode = self.createmode
17728
004bd533880d store: invoke "os.path.isdir()" via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17727
diff changeset
   966
        self.rawvfs = vfs
31234
9b7a2ef4f27c vfs: use 'vfs' module directly in 'mercurial.store'
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31219
diff changeset
   967
        self.vfs = vfsmod.filtervfs(vfs, encodefilename)
17653
dacb50696b75 store: initialize "vfs" fields by "vfs" constructors
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17652
diff changeset
   968
        self.opener = self.vfs
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   969
50490
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   970
    def _walk(self, relpath, recurse, undecodable=None):
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   971
        old = super()._walk(relpath, recurse)
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   972
        new = []
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   973
        for f1, value in old:
6892
dab95717058d verify: check repo.store
Adrian Buehlmann <adrian@cadifra.com>
parents: 6890
diff changeset
   974
            try:
47877
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   975
                f2 = decodefilename(f1)
6892
dab95717058d verify: check repo.store
Adrian Buehlmann <adrian@cadifra.com>
parents: 6890
diff changeset
   976
            except KeyError:
47877
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   977
                if undecodable is None:
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   978
                    msg = _(b'undecodable revlog name %s') % f1
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   979
                    raise error.StorageError(msg)
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   980
                else:
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   981
                    undecodable.append(f1)
2174f54aab18 store: return just one filename in walk functions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>
parents: 47785
diff changeset
   982
                    continue
50490
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   983
            new.append((f2, value))
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   984
        return new
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   985
50501
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
   986
    def data_entries(
50490
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   987
        self, matcher=None, undecodable=None
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   988
    ) -> Generator[BaseStoreEntry, None, None]:
50501
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
   989
        entries = super(encodedstore, self).data_entries(
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
   990
            undecodable=undecodable
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
   991
        )
50490
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   992
        for entry in entries:
50491
b4953fad744e store: do the revlog matching on entry directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50490
diff changeset
   993
            if _match_tracked_entry(entry, matcher):
50490
816e8bc6e066 store: split the wrapping of encodedstore between _wrap and datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50489
diff changeset
   994
                yield entry
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   995
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   996
    def join(self, f):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   997
        return self.path + b'/' + encodefilename(f)
6840
80e51429cb9a introduce store classes
Adrian Buehlmann <adrian@cadifra.com>
parents: 6839
diff changeset
   998
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
   999
    def copylist(self):
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1000
        return [b'requires', b'00changelog.i'] + [b'store/' + f for f in _data]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1001
6903
0642d9d7ec80 clone: get a list of files to clone from store
Matt Mackall <mpm@selenic.com>
parents: 6902
diff changeset
  1002
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48941
diff changeset
  1003
class fncache:
8531
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
  1004
    # the filename used to be partially encoded
810387f59696 filelog encoding: move the encoding/decoding into store
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8530
diff changeset
  1005
    # hence the encodedir/decodedir dance
17722
3b976051034d store: rename field name from "opener" to "vfs" in internal classes for fncache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17721
diff changeset
  1006
    def __init__(self, vfs):
3b976051034d store: rename field name from "opener" to "vfs" in internal classes for fncache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17721
diff changeset
  1007
        self.vfs = vfs
50315
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
  1008
        self._ignores = set()
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1009
        self.entries = None
13391
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
  1010
        self._dirty = False
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1011
        # set of new additions to fncache
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1012
        self.addls = set()
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1013
52169
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1014
    @property
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1015
    def is_loaded(self):
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1016
        return self.entries is not None
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1017
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1018
    def ensureloaded(self, warn=None):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
  1019
        """read the fncache file if not already read.
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1020
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1021
        If the file on disk is corrupted, raise. If warn is provided,
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45483
diff changeset
  1022
        warn and keep going instead."""
52169
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1023
        if not self.is_loaded:
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1024
            self._load(warn)
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1025
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1026
    def _load(self, warn=None):
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1027
        '''fill the entries from the fncache file'''
13391
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
  1028
        self._dirty = False
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1029
        try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1030
            fp = self.vfs(b'fncache', mode=b'rb')
52640
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 52639
diff changeset
  1031
        except OSError:
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1032
            # skip nonexistent file
16404
9fca5b056c0a store: speed up read and write of large fncache files
Bryan O'Sullivan <bryano@fb.com>
parents: 15742
diff changeset
  1033
            self.entries = set()
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1034
            return
41978
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1035
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1036
        self.entries = set()
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1037
        chunk = b''
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1038
        for c in iter(functools.partial(fp.read, fncache_chunksize), b''):
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1039
            chunk += c
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1040
            try:
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1041
                p = chunk.rindex(b'\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1042
                self.entries.update(decodedir(chunk[: p + 1]).splitlines())
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1043
                chunk = chunk[p + 1 :]
41978
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1044
            except ValueError:
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1045
                # substring '\n' not found, maybe the entry is bigger than the
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1046
                # chunksize, so let's keep iterating
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1047
                pass
a56487081109 store: don't read the whole fncache in memory
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41973
diff changeset
  1048
41981
a920a9e1795a store: error out if fncache does not ends with a newline
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41978
diff changeset
  1049
        if chunk:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1050
            msg = _(b"fncache does not ends with a newline")
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1051
            if warn:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1052
                warn(msg + b'\n')
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1053
            else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1054
                raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1055
                    msg,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1056
                    hint=_(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1057
                        b"use 'hg debugrebuildfncache' to "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1058
                        b"rebuild the fncache"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1059
                    ),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1060
                )
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1061
        self._checkentries(fp, warn)
41973
d7ef84e595f8 store: move logic to check for invalid entry in fncache to own function
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41093
diff changeset
  1062
        fp.close()
d7ef84e595f8 store: move logic to check for invalid entry in fncache to own function
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 41093
diff changeset
  1063
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1064
    def _checkentries(self, fp, warn):
47062
f38bf44e077f black: make codebase compatible with black v21.4b2 and v20.8b1
Kyle Lippincott <spectral@google.com>
parents: 46990
diff changeset
  1065
        """make sure there is no empty string in entries"""
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1066
        if b'' in self.entries:
16404
9fca5b056c0a store: speed up read and write of large fncache files
Bryan O'Sullivan <bryano@fb.com>
parents: 15742
diff changeset
  1067
            fp.seek(0)
48941
fd5b8e696b75 py3: stop using util.iterfile()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48913
diff changeset
  1068
            for n, line in enumerate(fp):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1069
                if not line.rstrip(b'\n'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1070
                    t = _(b'invalid entry in fncache, line %d') % (n + 1)
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1071
                    if warn:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1072
                        warn(t + b'\n')
42715
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1073
                    else:
f59f8a5e9096 fncache: make debugrebuildfncache not fail on broken fncache
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 42341
diff changeset
  1074
                        raise error.Abort(t)
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1075
20883
cd443c7589cc fncache: move fncache writing to be in a transaction
Durham Goode <durham@fb.com>
parents: 20879
diff changeset
  1076
    def write(self, tr):
16404
9fca5b056c0a store: speed up read and write of large fncache files
Bryan O'Sullivan <bryano@fb.com>
parents: 15742
diff changeset
  1077
        if self._dirty:
52169
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1078
            assert self.is_loaded
40748
df8ed31a8ad8 store: write fncache only once if there are both adds and removes
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40736
diff changeset
  1079
            self.entries = self.entries | self.addls
df8ed31a8ad8 store: write fncache only once if there are both adds and removes
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40736
diff changeset
  1080
            self.addls = set()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1081
            tr.addbackup(b'fncache')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1082
            fp = self.vfs(b'fncache', mode=b'wb', atomictemp=True)
20879
cd03854a2e06 fncache: remove the rewriting logic
Durham Goode <durham@fb.com>
parents: 19903
diff changeset
  1083
            if self.entries:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1084
                fp.write(encodedir(b'\n'.join(self.entries) + b'\n'))
20879
cd03854a2e06 fncache: remove the rewriting logic
Durham Goode <durham@fb.com>
parents: 19903
diff changeset
  1085
            fp.close()
cd03854a2e06 fncache: remove the rewriting logic
Durham Goode <durham@fb.com>
parents: 19903
diff changeset
  1086
            self._dirty = False
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1087
        if self.addls:
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1088
            # if we have just new entries, let's append them to the fncache
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1089
            tr.addbackup(b'fncache')
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1090
            fp = self.vfs(b'fncache', mode=b'ab', atomictemp=True)
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1091
            if self.addls:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1092
                fp.write(encodedir(b'\n'.join(self.addls) + b'\n'))
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1093
            fp.close()
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1094
            self.entries = None
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1095
            self.addls = set()
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1096
50315
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
  1097
    def addignore(self, fn):
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
  1098
        self._ignores.add(fn)
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
  1099
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1100
    def add(self, fn):
50315
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
  1101
        if fn in self._ignores:
cf6e1d535602 fncache: make it possible to ignore some file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49306
diff changeset
  1102
            return
52169
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1103
        if not self.is_loaded:
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1104
            self._load()
10577
d5bd1beff794 store: only add new entries to the fncache file
Adrian Buehlmann <adrian@cadifra.com>
parents: 10339
diff changeset
  1105
        if fn not in self.entries:
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1106
            self.addls.add(fn)
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1107
20885
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1108
    def remove(self, fn):
52169
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1109
        if not self.is_loaded:
20885
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1110
            self._load()
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1111
        if fn in self.addls:
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1112
            self.addls.remove(fn)
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1113
            return
20885
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1114
        try:
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1115
            self.entries.remove(fn)
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1116
            self._dirty = True
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1117
        except KeyError:
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1118
            pass
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1119
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
  1120
    def __contains__(self, fn):
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1121
        if fn in self.addls:
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1122
            return True
52169
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1123
        if not self.is_loaded:
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1124
            self._load()
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
  1125
        return fn in self.entries
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1126
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1127
    def __iter__(self):
52169
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1128
        if not self.is_loaded:
8530
03196ac9a8b9 store: refactor the fncache handling
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8480
diff changeset
  1129
            self._load()
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1130
        return iter(self.entries | self.addls)
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1131
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1132
41093
6498f0e03526 vfs: fix proxyvfs inheritance
Boris Feld <boris.feld@octobus.net>
parents: 40748
diff changeset
  1133
class _fncachevfs(vfsmod.proxyvfs):
17721
cf236e3501c3 store: rename argument name from "op"(ener) to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17693
diff changeset
  1134
    def __init__(self, vfs, fnc, encode):
33412
a42369e04aee vfs: rename auditvfs to proxyvfs
Yuya Nishihara <yuya@tcha.org>
parents: 32372
diff changeset
  1135
        vfsmod.proxyvfs.__init__(self, vfs)
52170
9fbdf355c916 fncachestore: add typing information
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52169
diff changeset
  1136
        self.fncache: fncache = fnc
14194
3a90fb3addc1 store: break up reference cycle introduced in 9cbff8a39a2a
Adrian Buehlmann <adrian@cadifra.com>
parents: 14166
diff changeset
  1137
        self.encode = encode
52171
bd43465af568 fncache: add attribute to check whether we're using dotencode
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52170
diff changeset
  1138
        self.uses_dotencode = encode is _pathencode
14194
3a90fb3addc1 store: break up reference cycle introduced in 9cbff8a39a2a
Adrian Buehlmann <adrian@cadifra.com>
parents: 14166
diff changeset
  1139
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1140
    def __call__(self, path, mode=b'r', *args, **kw):
38661
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
  1141
        encoded = self.encode(path)
50464
0dbab42adca5 store: do not record file that are not part of a revlog in fncache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50448
diff changeset
  1142
        if (
0dbab42adca5 store: do not record file that are not part of a revlog in fncache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50448
diff changeset
  1143
            mode not in (b'r', b'rb')
0dbab42adca5 store: do not record file that are not part of a revlog in fncache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50448
diff changeset
  1144
            and (path.startswith(b'data/') or path.startswith(b'meta/'))
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
  1145
            and is_revlog_file(path)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1146
        ):
38661
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
  1147
            # do not trigger a fncache load when adding a file that already is
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
  1148
            # known to exist.
52169
ba9df4e6abb8 fncache: refactor load check into a property
Rapha?l Gom?s <rgomes@octobus.net>
parents: 52066
diff changeset
  1149
            notload = not self.fncache.is_loaded and (
50809
30381812708d fncache: fix a bug that corrupts the fncache after transaction rollback
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50703
diff changeset
  1150
                # if the file has size zero, it should be considered as missing.
30381812708d fncache: fix a bug that corrupts the fncache after transaction rollback
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50703
diff changeset
  1151
                # Such zero-size files are the result of truncation when a
30381812708d fncache: fix a bug that corrupts the fncache after transaction rollback
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50703
diff changeset
  1152
                # transaction is aborted.
30381812708d fncache: fix a bug that corrupts the fncache after transaction rollback
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50703
diff changeset
  1153
                self.vfs.exists(encoded)
30381812708d fncache: fix a bug that corrupts the fncache after transaction rollback
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50703
diff changeset
  1154
                and self.vfs.stat(encoded).st_size
30381812708d fncache: fix a bug that corrupts the fncache after transaction rollback
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 50703
diff changeset
  1155
            )
38661
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
  1156
            if not notload:
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
  1157
                self.fncache.add(path)
8ac0c9cd4c48 fncache: avoid loading the filename cache when not actually modifying it
Martijn Pieters <mj@zopatista.com>
parents: 37409
diff changeset
  1158
        return self.vfs(encoded, mode, *args, **kw)
14194
3a90fb3addc1 store: break up reference cycle introduced in 9cbff8a39a2a
Adrian Buehlmann <adrian@cadifra.com>
parents: 14166
diff changeset
  1159
51878
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
  1160
    def join(self, path: Optional[bytes], *insidef: bytes) -> bytes:
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
  1161
        insidef = (self.encode(f) for f in insidef)
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
  1162
17725
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17724
diff changeset
  1163
        if path:
51878
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
  1164
            return self.vfs.join(self.encode(path), *insidef)
17725
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17724
diff changeset
  1165
        else:
51878
2391a5fa111e store: fix a signature mismatch for a vfs subclass
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
  1166
            return self.vfs.join(path, *insidef)
17725
ffd589d4b785 vfs: define "join()" in each classes derived from "abstractvfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17724
diff changeset
  1167
47443
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
  1168
    def register_file(self, path):
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
  1169
        """generic hook point to lets fncache steer its stew"""
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
  1170
        if path.startswith(b'data/') or path.startswith(b'meta/'):
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
  1171
            self.fncache.add(path)
9ab54aa56982 vfs: add a `register_file` method on the vfs class
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47389
diff changeset
  1172
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1173
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1174
class fncachestore(basicstore):
17651
3b49c28658f6 store: rename "openertype" argument to "vfstype"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17649
diff changeset
  1175
    def __init__(self, path, vfstype, dotencode):
17591
9a5c2ecd1158 store: move encode lambda logic into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17590
diff changeset
  1176
        if dotencode:
18435
8c019d2fd7c0 store: switch to C-based hashed path encoding
Bryan O'Sullivan <bryano@fb.com>
parents: 18430
diff changeset
  1177
            encode = _pathencode
17591
9a5c2ecd1158 store: move encode lambda logic into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17590
diff changeset
  1178
        else:
9a5c2ecd1158 store: move encode lambda logic into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17590
diff changeset
  1179
            encode = _plainhybridencode
12687
34d8247a4595 store: encode first period or space in filenames (issue1713)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12171
diff changeset
  1180
        self.encode = encode
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1181
        vfs = vfstype(path + b'/store')
17724
bf4b72d8dd4d store: initialize vfs field first to use it for initialization of others
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17722
diff changeset
  1182
        self.path = vfs.base
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1183
        self.pathsep = self.path + b'/'
17726
7cb7e17c23b2 store: invoke "os.stat()" for "createmode" initialization via vfs
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17725
diff changeset
  1184
        self.createmode = _calcmode(vfs)
17652
2c6f7231becc store: rename "op" variables to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17651
diff changeset
  1185
        vfs.createmode = self.createmode
17727
6492b39a44d5 store: replace invocation of "getsize()" by "vfs.stat()"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17726
diff changeset
  1186
        self.rawvfs = vfs
17652
2c6f7231becc store: rename "op" variables to "vfs"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17651
diff changeset
  1187
        fnc = fncache(vfs)
9133
996c1cd8f530 store: eliminate reference cycle in fncachestore
Simon Heimberg <simohe@besonet.ch>
parents: 8778
diff changeset
  1188
        self.fncache = fnc
17653
dacb50696b75 store: initialize "vfs" fields by "vfs" constructors
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17652
diff changeset
  1189
        self.vfs = _fncachevfs(vfs, fnc, encode)
dacb50696b75 store: initialize "vfs" fields by "vfs" constructors
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 17652
diff changeset
  1190
        self.opener = self.vfs
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1191
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1192
    def join(self, f):
17562
b42b0729744d store: reduce string concatenation when joining
Bryan O'Sullivan <bryano@fb.com>
parents: 17555
diff changeset
  1193
        return self.pathsep + self.encode(f)
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1194
17731
c85dbae29684 store: restore getsize method
Matt Mackall <mpm@selenic.com>
parents: 17728
diff changeset
  1195
    def getsize(self, path):
c85dbae29684 store: restore getsize method
Matt Mackall <mpm@selenic.com>
parents: 17728
diff changeset
  1196
        return self.rawvfs.stat(path).st_size
c85dbae29684 store: restore getsize method
Matt Mackall <mpm@selenic.com>
parents: 17728
diff changeset
  1197
50501
862e3a13da44 store: rename `datafiles` to `data_entries`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50495
diff changeset
  1198
    def data_entries(
50468
521fec115dad store: use a StoreEntry object instead of tuple for store files
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50466
diff changeset
  1199
        self, matcher=None, undecodable=None
50470
5a2fb64d38b2 store: use specialized class for store entries
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50469
diff changeset
  1200
    ) -> Generator[BaseStoreEntry, None, None]:
50477
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50476
diff changeset
  1201
        # Note: all files in fncache should be revlog related, However the
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50476
diff changeset
  1202
        # fncache might contains such file added by previous version of
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50476
diff changeset
  1203
        # Mercurial.
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
  1204
        files = ((f, None) for f in self.fncache if is_revlog_file(f))
50477
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50476
diff changeset
  1205
        by_revlog = _gather_revlog(files)
d4f54aded22e store: also group files by revlog in fncache version of datafiles
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50476
diff changeset
  1206
        for revlog, details in by_revlog:
50479
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
  1207
            if revlog.startswith(b'data/'):
50677
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
  1208
                rl_type = KIND_FILELOG
50480
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
  1209
                revlog_target_id = revlog.split(b'/', 1)[1]
50479
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
  1210
            elif revlog.startswith(b'meta/'):
50677
e324329fc97e store: use the revlog type from revlog's constant instead of custom one
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50669
diff changeset
  1211
                rl_type = KIND_MANIFESTLOG
50480
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
  1212
                # drop the initial directory and the `00manifest` file part
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
  1213
                tmp = revlog.split(b'/', 1)[1]
60e613f6a229 store: add a `target_id` attribute on RevlogStoreEntry
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50479
diff changeset
  1214
                revlog_target_id = tmp.rsplit(b'/', 1)[0] + b'/'
50479
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
  1215
            else:
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
  1216
                # unreachable
1fc25227b068 store: actually tag tree manifest revlogs as manifest revlogs
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50478
diff changeset
  1217
                assert False, revlog
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
  1218
            entry = RevlogStoreEntry(
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
  1219
                path_prefix=revlog,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
  1220
                revlog_type=rl_type,
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
  1221
                target_id=revlog_target_id,
50668
e06d1a779eb6 store: stop relying on a `revlog_type` property
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50667
diff changeset
  1222
                details=details,
50493
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
  1223
            )
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
  1224
            if _match_tracked_entry(entry, matcher):
e50d1fe7ebb4 store: issue a single entry for each revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50492
diff changeset
  1225
                yield entry
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1226
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1227
    def copylist(self):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1228
        d = (
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1229
            b'bookmarks',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1230
            b'narrowspec',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1231
            b'data',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1232
            b'meta',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1233
            b'dh',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1234
            b'fncache',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1235
            b'phaseroots',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1236
            b'obsstore',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1237
            b'00manifest.d',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1238
            b'00manifest.i',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1239
            b'00changelog.d',
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1240
            b'00changelog.i',
45483
d252f51ab032 share: introduce config option to store requires in .hg/store
Pulkit Goyal <7895pulkit@gmail.com>
parents: 45351
diff changeset
  1241
            b'requires',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42905
diff changeset
  1242
        )
45351
909dafff6a78 store: refactor space delimited list to proper data structure
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
  1243
        return [b'requires', b'00changelog.i'] + [b'store/' + f for f in d]
7229
7946503ec76e introduce fncache repository layout
Adrian Buehlmann <adrian@cadifra.com>
parents: 6989
diff changeset
  1244
20883
cd443c7589cc fncache: move fncache writing to be in a transaction
Durham Goode <durham@fb.com>
parents: 20879
diff changeset
  1245
    def write(self, tr):
cd443c7589cc fncache: move fncache writing to be in a transaction
Durham Goode <durham@fb.com>
parents: 20879
diff changeset
  1246
        self.fncache.write(tr)
13391
d00bbff8600e fncachestore: defer updating the fncache file to a single file open
Adrian Buehlmann <adrian@cadifra.com>
parents: 13169
diff changeset
  1247
20884
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
  1248
    def invalidatecaches(self):
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
  1249
        self.fncache.entries = None
40736
0728d87a8631 store: append to fncache if there are only new files to write
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 40624
diff changeset
  1250
        self.fncache.addls = set()
20884
2efdd186925d caches: invalidate store caches when lock is taken
Durham Goode <durham@fb.com>
parents: 20883
diff changeset
  1251
20885
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1252
    def markremoved(self, fn):
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1253
        self.fncache.remove(fn)
f49d60fa40a5 fncache: clean up fncache during strips
Durham Goode <durham@fb.com>
parents: 20884
diff changeset
  1254
17783
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
  1255
    def _exists(self, f):
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
  1256
        ef = self.encode(f)
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
  1257
        try:
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
  1258
            self.getsize(ef)
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
  1259
            return True
49306
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49284
diff changeset
  1260
        except FileNotFoundError:
17783
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
  1261
            return False
df55ce6854c3 store: add new _exists helper function on fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17782
diff changeset
  1262
17745
b9a56b816ff2 store: add a contains method to fncachestore
smuralid
parents: 17744
diff changeset
  1263
    def __contains__(self, path):
b9a56b816ff2 store: add a contains method to fncachestore
smuralid
parents: 17744
diff changeset
  1264
        '''Checks if the store contains path'''
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1265
        path = b"/".join((b"data", path))
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
  1266
        # check for files (exact match)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1267
        e = path + b'.i'
17784
73e1ab39792c store: fncache may contain non-existent entries (fixes b9a56b816ff2)
Adrian Buehlmann <adrian@cadifra.com>
parents: 17783
diff changeset
  1268
        if e in self.fncache and self._exists(e):
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
  1269
            return True
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
  1270
        # now check for directories (prefix match)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1271
        if not path.endswith(b'/'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1272
            path += b'/'
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
  1273
        for e in self.fncache:
17784
73e1ab39792c store: fncache may contain non-existent entries (fixes b9a56b816ff2)
Adrian Buehlmann <adrian@cadifra.com>
parents: 17783
diff changeset
  1274
            if e.startswith(path) and self._exists(e):
17782
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
  1275
                return True
8095306c1fb2 store: move __contains__() implementation from class fncache into fncachestore
Adrian Buehlmann <adrian@cadifra.com>
parents: 17747
diff changeset
  1276
        return False