contrib/pull_logger.py
author pacien <pacien.trangirard@pacien.net>
Mon, 25 Jul 2022 22:47:15 +0200
changeset 49508 791050360486
child 49509 946c023212b8
permissions -rw-r--r--
contrib: add pull_logger extension This extension logs the pull parameters, i.e. the remote and common heads, when pulling from the local repository. The collected data should give an idea of the state of a pair of repositories and allow replaying past synchronisations between them. This is particularly useful for working on data exchange, bundling and caching-related optimisations.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
49508
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
     1
# pull_logger.py - Logs pulls to a JSON-line file in the repo's VFS.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
     2
#
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
     3
# Copyright 2022  Pacien TRAN-GIRARD <pacien.trangirard@pacien.net>
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
     4
#
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
     5
# This software may be used and distributed according to the terms of the
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
     6
# GNU General Public License version 2 or any later version.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
     7
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
     8
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
     9
'''logs pull parameters to a file
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    10
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    11
This extension logs the pull parameters, i.e. the remote and common heads,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    12
when pulling from the local repository.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    13
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    14
The collected data should give an idea of the state of a pair of repositories
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    15
and allow replaying past synchronisations between them. This is particularly
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    16
useful for working on data exchange, bundling and caching-related
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    17
optimisations.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    18
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    19
The record is a JSON-line file located in the repository's VFS at
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    20
.hg/pull_log.jsonl.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    21
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    22
Log write failures are not considered fatal: log writes may be skipped for any
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    23
reason such as insufficient storage or a timeout.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    24
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    25
The timeouts of the exclusive lock used when writing to the lock file can be
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    26
configured through the 'timeout.lock' and 'timeout.warn' options of this
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    27
plugin. Those are not expected to be held for a significant time in practice.::
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    28
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    29
  [pull-logger]
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    30
  timeout.lock = 300
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    31
  timeout.warn = 100
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    32
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    33
Note: there is no automatic log rotation and the size of the log is not capped.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    34
'''
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    35
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    36
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    37
import json
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    38
import time
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    39
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    40
from mercurial.i18n import _
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    41
from mercurial.utils import stringutil
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    42
from mercurial import (
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    43
    error,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    44
    extensions,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    45
    lock,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    46
    registrar,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    47
    wireprotov1server,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    48
)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    49
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    50
EXT_NAME = b'pull-logger'
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    51
EXT_VERSION_CODE = 0
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    52
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    53
LOG_FILE = b'pull_log.jsonl'
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    54
LOCK_NAME = LOG_FILE + b'.lock'
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    55
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    56
configtable = {}
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    57
configitem = registrar.configitem(configtable)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    58
configitem(EXT_NAME, b'timeout.lock', default=600)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    59
configitem(EXT_NAME, b'timeout.warn', default=120)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    60
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    61
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    62
def wrap_getbundle(orig, repo, proto, others, *args, **kwargs):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    63
    heads, common = extract_pull_heads(others)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    64
    log_entry = {
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    65
        'timestamp': time.time(),
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    66
        'logger_version': EXT_VERSION_CODE,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    67
        'heads': sorted(heads),
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    68
        'common': sorted(common),
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    69
    }
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    70
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    71
    try:
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    72
        write_to_log(repo, log_entry)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    73
    except (IOError, error.LockError) as err:
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    74
        msg = stringutil.forcebytestr(err)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    75
        repo.ui.warn(_(b'unable to append to pull log: %s\n') % msg)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    76
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    77
    return orig(repo, proto, others, *args, **kwargs)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    78
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    79
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    80
def extract_pull_heads(bundle_args):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    81
    opts = wireprotov1server.options(
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    82
        b'getbundle',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    83
        wireprotov1server.wireprototypes.GETBUNDLE_ARGUMENTS.keys(),
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    84
        bundle_args.copy(),  # this call consumes the args destructively
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    85
    )
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    86
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    87
    heads = opts.get(b'heads', b'').decode('utf-8').split(' ')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    88
    common = opts.get(b'common', b'').decode('utf-8').split(' ')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    89
    return (heads, common)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    90
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    91
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    92
def write_to_log(repo, entry):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    93
    locktimeout = repo.ui.configint(EXT_NAME, b'timeout.lock')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    94
    lockwarntimeout = repo.ui.configint(EXT_NAME, b'timeout.warn')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    95
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    96
    with lock.trylock(
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    97
        ui=repo.ui,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    98
        vfs=repo.vfs,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
    99
        lockname=LOCK_NAME,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   100
        timeout=locktimeout,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   101
        warntimeout=lockwarntimeout,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   102
    ):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   103
        with repo.vfs.open(LOG_FILE, b'a+') as logfile:
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   104
            serialised = json.dumps(entry, sort_keys=True)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   105
            logfile.write(serialised.encode('utf-8'))
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   106
            logfile.write(b'\n')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   107
            logfile.flush()
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   108
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   109
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   110
def reposetup(ui, repo):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   111
    if repo.local():
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   112
        repo._wlockfreeprefix.add(LOG_FILE)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   113
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   114
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   115
def uisetup(ui):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   116
    del wireprotov1server.commands[b'getbundle']
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   117
    decorator = wireprotov1server.wireprotocommand(
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   118
        name=b'getbundle',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   119
        args=b'*',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   120
        permission=b'pull',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   121
    )
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   122
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   123
    extensions.wrapfunction(
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   124
        container=wireprotov1server,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   125
        funcname='getbundle',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   126
        wrapper=wrap_getbundle,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   127
    )
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   128
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
   129
    decorator(wireprotov1server.getbundle)