annotate mercurial/pushkey.py @ 37060:2ec1fb9de638

wireproto: add request IDs to frames One of my primary goals with the new wire protocol is to make operations faster and enable both client and server-side operations to scale to multiple CPU cores. One of the ways we make server interactions faster is by reducing the number of round trips to that server. With the existing wire protocol, the "batch" command facilitates executing multiple commands from a single request payload. The way it works is the requests for multiple commands are serialized. The server executes those commands sequentially then serializes all their results. As an optimization for reducing round trips, this is very effective. The technical implementation, however, is pretty bad and suffers from a number of deficiencies. For example, it creates a new place where authorization to run a command must be checked. (The lack of this checking in older Mercurial releases was CVE-2018-1000132.) The principles behind the "batch" command are sound. However, the execution is not. Therefore, I want to ditch "batch" in the new wire protocol and have protocol level support for issuing multiple requests in a single round trip. This commit introduces support in the frame-based wire protocol to facilitate this. We do this by adding a "request ID" to each frame. If a server sees frames associated with different "request IDs," it handles them as separate requests. All of this happening possibly as part of the same message from client to server (the same request body in the case of HTTP). We /could/ model the exchange the way pipelined HTTP requests do, where the server processes requests in order they are issued and received. But this artifically constrains scalability. A better model is to allow multi-requests to be executed concurrently and for responses to be sent and handled concurrently. So the specification explicitly allows this. There is some work to be done around specifying dependencies between multi-requests. We take the easy road for now and punt on this problem, declaring that if order is important, clients must not issue the request until responses to dependent requests have been received. This commit focuses on the boilerplate of implementing the request ID. The server reactor still can't manage multiple, in-flight request IDs. This will be addressed in a subsequent commit. Because the wire semantics have changed, we bump the version of the media type. Differential Revision: https://phab.mercurial-scm.org/D2869
author Gregory Szorc <gregory.szorc@gmail.com>
date Wed, 14 Mar 2018 16:51:34 -0700
parents 7b200566e474
children 57875cf423c9
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11367
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
1 # pushkey.py - dispatching for pushing and pulling keys
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
2 #
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
3 # Copyright 2010 Matt Mackall <mpm@selenic.com>
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
4 #
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
7
25969
7b200566e474 pushkey: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22953
diff changeset
8 from __future__ import absolute_import
7b200566e474 pushkey: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22953
diff changeset
9
7b200566e474 pushkey: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22953
diff changeset
10 from . import (
7b200566e474 pushkey: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22953
diff changeset
11 bookmarks,
7b200566e474 pushkey: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22953
diff changeset
12 encoding,
7b200566e474 pushkey: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22953
diff changeset
13 obsolete,
7b200566e474 pushkey: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22953
diff changeset
14 phases,
7b200566e474 pushkey: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22953
diff changeset
15 )
13353
689bf32b3bbd bookmarks: move pushkey functions into core
Matt Mackall <mpm@selenic.com>
parents: 11367
diff changeset
16
11367
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
17 def _nslist(repo):
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
18 n = {}
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
19 for k in _namespaces:
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
20 n[k] = ""
22953
b1d694d3975e obsolete: add exchange option
Durham Goode <durham@fb.com>
parents: 21661
diff changeset
21 if not obsolete.isenabled(repo, obsolete.exchangeopt):
17298
59c14bf5a48c pushkey: do not exchange obsole markers if feature is disabled
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 17075
diff changeset
22 n.pop('obsolete')
11367
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
23 return n
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
24
13353
689bf32b3bbd bookmarks: move pushkey functions into core
Matt Mackall <mpm@selenic.com>
parents: 11367
diff changeset
25 _namespaces = {"namespaces": (lambda *x: False, _nslist),
15648
79cc89de5be1 phases: add basic pushkey support
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 13353
diff changeset
26 "bookmarks": (bookmarks.pushbookmark, bookmarks.listbookmarks),
79cc89de5be1 phases: add basic pushkey support
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 13353
diff changeset
27 "phases": (phases.pushphase, phases.listphases),
17075
28ed1c4511ce obsolete: exchange obsolete marker over pushkey
Pierre-Yves.David@ens-lyon.org
parents: 15648
diff changeset
28 "obsolete": (obsolete.pushmarker, obsolete.listmarkers),
15648
79cc89de5be1 phases: add basic pushkey support
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 13353
diff changeset
29 }
11367
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
30
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
31 def register(namespace, pushkey, listkeys):
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
32 _namespaces[namespace] = (pushkey, listkeys)
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
33
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
34 def _get(namespace):
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
35 return _namespaces.get(namespace, (lambda *x: False, lambda *x: {}))
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
36
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
37 def push(repo, namespace, key, old, new):
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
38 '''should succeed iff value was old'''
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
39 pk = _get(namespace)[0]
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
40 return pk(repo, key, old, new)
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
41
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
42 def list(repo, namespace):
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
43 '''return a dict'''
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
44 lk = _get(namespace)[1]
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
45 return lk(repo)
ca4fc993087c pushkey: add pushkey core
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
46
21661
2f52a16f2bee pushkey: add an ``encode`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21659
diff changeset
47 encode = encoding.fromlocal
2f52a16f2bee pushkey: add an ``encode`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21659
diff changeset
48
21659
a319842539f5 pushkey: add a ``decode`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21652
diff changeset
49 decode = encoding.tolocal
a319842539f5 pushkey: add a ``decode`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21652
diff changeset
50
21650
a2c7ae21e8f4 pushkey: introduce an ``encodekeys`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 17298
diff changeset
51 def encodekeys(keys):
a2c7ae21e8f4 pushkey: introduce an ``encodekeys`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 17298
diff changeset
52 """encode the content of a pushkey namespace for exchange over the wire"""
21661
2f52a16f2bee pushkey: add an ``encode`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21659
diff changeset
53 return '\n'.join(['%s\t%s' % (encode(k), encode(v)) for k, v in keys])
21652
ed6e61eaebc0 pushkey: introduce an ``decodekeys`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21650
diff changeset
54
ed6e61eaebc0 pushkey: introduce an ``decodekeys`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21650
diff changeset
55 def decodekeys(data):
ed6e61eaebc0 pushkey: introduce an ``decodekeys`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21650
diff changeset
56 """decode the content of a pushkey namespace from exchange over the wire"""
ed6e61eaebc0 pushkey: introduce an ``decodekeys`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21650
diff changeset
57 result = {}
ed6e61eaebc0 pushkey: introduce an ``decodekeys`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21650
diff changeset
58 for l in data.splitlines():
ed6e61eaebc0 pushkey: introduce an ``decodekeys`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21650
diff changeset
59 k, v = l.split('\t')
21659
a319842539f5 pushkey: add a ``decode`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21652
diff changeset
60 result[decode(k)] = decode(v)
21652
ed6e61eaebc0 pushkey: introduce an ``decodekeys`` function
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 21650
diff changeset
61 return result