Mercurial > public > mercurial-scm > hg

annotate contrib/pull_logger.py @ 49508:791050360486

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
contrib: add pull_logger extension This extension logs the pull parameters, i.e. the remote and common heads, when pulling from the local repository. The collected data should give an idea of the state of a pair of repositories and allow replaying past synchronisations between them. This is particularly useful for working on data exchange, bundling and caching-related optimisations.
author pacien <pacien.trangirard@pacien.net>
date Mon, 25 Jul 2022 22:47:15 +0200
parents
children 946c023212b8
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
49508
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
1 # pull_logger.py - Logs pulls to a JSON-line file in the repo's VFS.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
2 #
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
3 # Copyright 2022 Pacien TRAN-GIRARD <pacien.trangirard@pacien.net>
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
4 #
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
7
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
8
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
9 '''logs pull parameters to a file
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
10
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
11 This extension logs the pull parameters, i.e. the remote and common heads,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
12 when pulling from the local repository.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
13
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
14 The collected data should give an idea of the state of a pair of repositories
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
15 and allow replaying past synchronisations between them. This is particularly
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
16 useful for working on data exchange, bundling and caching-related
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
17 optimisations.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
18
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
19 The record is a JSON-line file located in the repository's VFS at
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
20 .hg/pull_log.jsonl.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
21
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
22 Log write failures are not considered fatal: log writes may be skipped for any
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
23 reason such as insufficient storage or a timeout.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
24
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
25 The timeouts of the exclusive lock used when writing to the lock file can be
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
26 configured through the 'timeout.lock' and 'timeout.warn' options of this
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
27 plugin. Those are not expected to be held for a significant time in practice.::
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
28
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
29 [pull-logger]
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
30 timeout.lock = 300
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
31 timeout.warn = 100
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
32
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
33 Note: there is no automatic log rotation and the size of the log is not capped.
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
34 '''
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
35
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
36
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
37 import json
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
38 import time
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
39
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
40 from mercurial.i18n import _
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
41 from mercurial.utils import stringutil
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
42 from mercurial import (
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
43 error,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
44 extensions,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
45 lock,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
46 registrar,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
47 wireprotov1server,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
48 )
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
49
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
50 EXT_NAME = b'pull-logger'
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
51 EXT_VERSION_CODE = 0
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
52
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
53 LOG_FILE = b'pull_log.jsonl'
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
54 LOCK_NAME = LOG_FILE + b'.lock'
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
55
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
56 configtable = {}
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
57 configitem = registrar.configitem(configtable)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
58 configitem(EXT_NAME, b'timeout.lock', default=600)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
59 configitem(EXT_NAME, b'timeout.warn', default=120)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
60
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
61
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
62 def wrap_getbundle(orig, repo, proto, others, *args, **kwargs):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
63 heads, common = extract_pull_heads(others)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
64 log_entry = {
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
65 'timestamp': time.time(),
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
66 'logger_version': EXT_VERSION_CODE,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
67 'heads': sorted(heads),
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
68 'common': sorted(common),
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
69 }
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
70
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
71 try:
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
72 write_to_log(repo, log_entry)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
73 except (IOError, error.LockError) as err:
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
74 msg = stringutil.forcebytestr(err)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
75 repo.ui.warn(_(b'unable to append to pull log: %s\n') % msg)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
76
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
77 return orig(repo, proto, others, *args, **kwargs)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
78
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
79
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
80 def extract_pull_heads(bundle_args):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
81 opts = wireprotov1server.options(
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
82 b'getbundle',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
83 wireprotov1server.wireprototypes.GETBUNDLE_ARGUMENTS.keys(),
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
84 bundle_args.copy(), # this call consumes the args destructively
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
85 )
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
86
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
87 heads = opts.get(b'heads', b'').decode('utf-8').split(' ')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
88 common = opts.get(b'common', b'').decode('utf-8').split(' ')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
89 return (heads, common)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
90
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
91
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
92 def write_to_log(repo, entry):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
93 locktimeout = repo.ui.configint(EXT_NAME, b'timeout.lock')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
94 lockwarntimeout = repo.ui.configint(EXT_NAME, b'timeout.warn')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
95
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
96 with lock.trylock(
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
97 ui=repo.ui,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
98 vfs=repo.vfs,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
99 lockname=LOCK_NAME,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
100 timeout=locktimeout,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
101 warntimeout=lockwarntimeout,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
102 ):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
103 with repo.vfs.open(LOG_FILE, b'a+') as logfile:
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
104 serialised = json.dumps(entry, sort_keys=True)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
105 logfile.write(serialised.encode('utf-8'))
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
106 logfile.write(b'\n')
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
107 logfile.flush()
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
108
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
109
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
110 def reposetup(ui, repo):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
111 if repo.local():
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
112 repo._wlockfreeprefix.add(LOG_FILE)
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
113
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
114
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
115 def uisetup(ui):
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
116 del wireprotov1server.commands[b'getbundle']
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
117 decorator = wireprotov1server.wireprotocommand(
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
118 name=b'getbundle',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
119 args=b'*',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
120 permission=b'pull',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
121 )
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
122
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
123 extensions.wrapfunction(
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
124 container=wireprotov1server,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
125 funcname='getbundle',
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
126 wrapper=wrap_getbundle,
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
127 )
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
128
791050360486 contrib: add pull_logger extension
pacien <pacien.trangirard@pacien.net>
parents:
diff changeset
129 decorator(wireprotov1server.getbundle)