annotate mercurial/hgweb/protocol.py @ 30475:2add671bf55b

wireproto: perform chunking and compression at protocol layer (API) Currently, the "streamres" response type is populated with a generator of chunks with compression possibly already applied. This puts the onus on commands to perform chunking and compression. Architecturally, I think this is the wrong place to perform this work. I think commands should say "here is the data" and the protocol layer should take care of encoding the final bytes to put on the wire. Additionally, upcoming commits will improve wire protocol support for compression. Having a central place for performing compression in the protocol transport layer will be easier than having to deal with compression at the commands layer. This commit refactors the "streamres" response type to accept either a generator or an object with "read." Additionally, the type now accepts a flag indicating whether the response is a "version 1 compressible" response. This basically identifies all commands currently performing compression. I could have used a special type for this, but a flag works just as well. The argument name foreshadows the introduction of wire protocol changes, hence the "v1." The code for chunking and compressing has been moved to the output generation function for each protocol transport. Some code has been inlined, resulting in the deletion of now unused methods.
author Gregory Szorc <gregory.szorc@gmail.com>
date Sun, 20 Nov 2016 13:50:45 -0800
parents 038547a14d85
children b3a9ef3d30e8
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5598
d534ba1c4eb4 separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
1 #
d534ba1c4eb4 separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
2 # Copyright 21 May 2005 - (c) 2005 Jake Edge <jake@edge2.net>
d534ba1c4eb4 separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
3 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
d534ba1c4eb4 separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
4 #
8225
46293a0c7e9f updated license to be explicit about GPL version 2
Martin Geisler <mg@lazybytes.net>
parents: 8109
diff changeset
5 # This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9713
diff changeset
6 # GNU General Public License version 2 or any later version.
5598
d534ba1c4eb4 separate the wire protocol commands from the user interface commands
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
7
27046
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
8 from __future__ import absolute_import
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
9
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
10 import cgi
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
11
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
12 from .common import (
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
13 HTTP_OK,
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
14 )
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
15
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
16 from .. import (
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
17 util,
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
18 wireproto,
37fcfe52c68c hgweb: use absolute_import
Yuya Nishihara <yuya@tcha.org>
parents: 20903
diff changeset
19 )
28861
86db5cb55d46 pycompat: switch to util.stringio for py3 compat
timeless <timeless@mozdev.org>
parents: 28530
diff changeset
20 stringio = util.stringio
5963
5be210afe1b8 hgweb: explicitly check if requested command exists
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 5915
diff changeset
21
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28861
diff changeset
22 urlerr = util.urlerr
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28861
diff changeset
23 urlreq = util.urlreq
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28861
diff changeset
24
5993
948a41e77902 hgweb: explicit response status
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 5963
diff changeset
25 HGTYPE = 'application/mercurial-0.1'
15017
f4522df38c65 wireproto: add out-of-band error class to allow remote repo to report errors
Andrew Pritchard <andrewp@fogcreek.com>
parents: 14614
diff changeset
26 HGERRTYPE = 'application/hg-error'
11595
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
27
20903
8d477543882b wireproto: introduce an abstractserverproto class
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 18352
diff changeset
28 class webproto(wireproto.abstractserverproto):
14614
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
29 def __init__(self, req, ui):
11595
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
30 self.req = req
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
31 self.response = ''
14614
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
32 self.ui = ui
11595
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
33 def getargs(self, args):
14093
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
34 knownargs = self._args()
11595
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
35 data = {}
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
36 keys = args.split()
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
37 for k in keys:
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
38 if k == '*':
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
39 star = {}
14093
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
40 for key in knownargs.keys():
13721
3458c15ab2f0 wireproto: fix handling of '*' args for HTTP and SSH
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 12704
diff changeset
41 if key != 'cmd' and key not in keys:
14093
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
42 star[key] = knownargs[key][0]
11595
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
43 data['*'] = star
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
44 else:
14093
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
45 data[k] = knownargs[k][0]
11595
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
46 return [data[k] for k in keys]
14093
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
47 def _args(self):
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
48 args = self.req.form.copy()
28530
fd2acc5046f6 http: support sending hgargs via POST body instead of in GET or headers
Augie Fackler <augie@google.com>
parents: 27046
diff changeset
49 postlen = int(self.req.env.get('HTTP_X_HGARGS_POST', 0))
fd2acc5046f6 http: support sending hgargs via POST body instead of in GET or headers
Augie Fackler <augie@google.com>
parents: 27046
diff changeset
50 if postlen:
fd2acc5046f6 http: support sending hgargs via POST body instead of in GET or headers
Augie Fackler <augie@google.com>
parents: 27046
diff changeset
51 args.update(cgi.parse_qs(
fd2acc5046f6 http: support sending hgargs via POST body instead of in GET or headers
Augie Fackler <augie@google.com>
parents: 27046
diff changeset
52 self.req.read(postlen), keep_blank_values=True))
fd2acc5046f6 http: support sending hgargs via POST body instead of in GET or headers
Augie Fackler <augie@google.com>
parents: 27046
diff changeset
53 return args
14093
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
54 chunks = []
14094
d10c6835497e http: minor tweaks to long arg handling
Matt Mackall <mpm@selenic.com>
parents: 14093
diff changeset
55 i = 1
14494
1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents: 14094
diff changeset
56 while True:
14094
d10c6835497e http: minor tweaks to long arg handling
Matt Mackall <mpm@selenic.com>
parents: 14093
diff changeset
57 h = self.req.env.get('HTTP_X_HGARG_' + str(i))
14093
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
58 if h is None:
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
59 break
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
60 chunks += [h]
14094
d10c6835497e http: minor tweaks to long arg handling
Matt Mackall <mpm@selenic.com>
parents: 14093
diff changeset
61 i += 1
14093
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
62 args.update(cgi.parse_qs(''.join(chunks), keep_blank_values=True))
ce99d887585f httprepo: long arguments support (issue2126)
Steven Brown <StevenGBrown@gmail.com>
parents: 13721
diff changeset
63 return args
11621
e46a8b2331a6 protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11618
diff changeset
64 def getfile(self, fp):
e46a8b2331a6 protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11618
diff changeset
65 length = int(self.req.env['CONTENT_LENGTH'])
e46a8b2331a6 protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11618
diff changeset
66 for s in util.filechunkiter(self.req, limit=length):
e46a8b2331a6 protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11618
diff changeset
67 fp.write(s)
e46a8b2331a6 protocol: shuffle server methods to group send methods
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11618
diff changeset
68 def redirect(self):
14614
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
69 self.oldio = self.ui.fout, self.ui.ferr
28861
86db5cb55d46 pycompat: switch to util.stringio for py3 compat
timeless <timeless@mozdev.org>
parents: 28530
diff changeset
70 self.ui.ferr = self.ui.fout = stringio()
14614
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
71 def restore(self):
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
72 val = self.ui.fout.getvalue()
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
73 self.ui.ferr, self.ui.fout = self.oldio
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
74 return val
30206
d105195436c0 wireproto: compress data from a generator
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30016
diff changeset
75
d105195436c0 wireproto: compress data from a generator
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30016
diff changeset
76 def compresschunks(self, chunks):
29799
b1809f5d7630 hgweb: document why we don't allow untrusted settings to control zlib
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29760
diff changeset
77 # Don't allow untrusted settings because disabling compression or
b1809f5d7630 hgweb: document why we don't allow untrusted settings to control zlib
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29760
diff changeset
78 # setting a very high compression level could lead to flooding
b1809f5d7630 hgweb: document why we don't allow untrusted settings to control zlib
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29760
diff changeset
79 # the server's network or CPU.
30368
38130a0bbb99 hgweb: use compression engine API for zlib compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30206
diff changeset
80 opts = {'level': self.ui.configint('server', 'zliblevel', -1)}
38130a0bbb99 hgweb: use compression engine API for zlib compression
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30206
diff changeset
81 return util.compengines['zlib'].compressstream(chunks, opts)
30206
d105195436c0 wireproto: compress data from a generator
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30016
diff changeset
82
11595
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
83 def _client(self):
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
84 return 'remote:%s:%s:%s' % (
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
85 self.req.env.get('wsgi.url_scheme') or 'http',
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28861
diff changeset
86 urlreq.quote(self.req.env.get('REMOTE_HOST', '')),
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28861
diff changeset
87 urlreq.quote(self.req.env.get('REMOTE_USER', '')))
11595
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
88
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
89 def iscmd(cmd):
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
90 return cmd in wireproto.commands
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
91
368cd5325348 protocol: move hgweb protocol support back into protocol.py
Matt Mackall <mpm@selenic.com>
parents: 11594
diff changeset
92 def call(repo, req, cmd):
14614
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
93 p = webproto(req, repo.ui)
11625
cdeb861335d5 protocol: wrap non-string protocol responses in classes
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11623
diff changeset
94 rsp = wireproto.dispatch(repo, p, cmd)
11626
2f8adc60e013 protocol: use generators instead of req.write() for hgweb stream responses
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11625
diff changeset
95 if isinstance(rsp, str):
18352
e33b9b92a200 hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents: 18346
diff changeset
96 req.respond(HTTP_OK, HGTYPE, body=rsp)
e33b9b92a200 hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents: 18346
diff changeset
97 return []
11626
2f8adc60e013 protocol: use generators instead of req.write() for hgweb stream responses
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11625
diff changeset
98 elif isinstance(rsp, wireproto.streamres):
30475
2add671bf55b wireproto: perform chunking and compression at protocol layer (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30376
diff changeset
99 if rsp.reader:
2add671bf55b wireproto: perform chunking and compression at protocol layer (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30376
diff changeset
100 gen = iter(lambda: rsp.reader.read(32768), '')
2add671bf55b wireproto: perform chunking and compression at protocol layer (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30376
diff changeset
101 else:
2add671bf55b wireproto: perform chunking and compression at protocol layer (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30376
diff changeset
102 gen = rsp.gen
2add671bf55b wireproto: perform chunking and compression at protocol layer (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30376
diff changeset
103
2add671bf55b wireproto: perform chunking and compression at protocol layer (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30376
diff changeset
104 if rsp.v1compressible:
2add671bf55b wireproto: perform chunking and compression at protocol layer (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30376
diff changeset
105 gen = p.compresschunks(gen)
2add671bf55b wireproto: perform chunking and compression at protocol layer (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30376
diff changeset
106
11626
2f8adc60e013 protocol: use generators instead of req.write() for hgweb stream responses
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11625
diff changeset
107 req.respond(HTTP_OK, HGTYPE)
30475
2add671bf55b wireproto: perform chunking and compression at protocol layer (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30376
diff changeset
108 return gen
11626
2f8adc60e013 protocol: use generators instead of req.write() for hgweb stream responses
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 11625
diff changeset
109 elif isinstance(rsp, wireproto.pushres):
14614
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
110 val = p.restore()
18346
6c2563b2c1c6 hgweb: use Content-Length for pushres
Mads Kiilerich <mads@kiilerich.com>
parents: 15017
diff changeset
111 rsp = '%d\n%s' % (rsp.res, val)
18352
e33b9b92a200 hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents: 18346
diff changeset
112 req.respond(HTTP_OK, HGTYPE, body=rsp)
e33b9b92a200 hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents: 18346
diff changeset
113 return []
12703
40bb5853fc4b wireproto: introduce pusherr() to deal with "unsynced changes" error
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11626
diff changeset
114 elif isinstance(rsp, wireproto.pusherr):
12704
ca6e2adc3e4d wireproto/http: drain the incoming bundle in case of errors
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 12703
diff changeset
115 # drain the incoming bundle
ca6e2adc3e4d wireproto/http: drain the incoming bundle in case of errors
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 12703
diff changeset
116 req.drain()
14614
afccc64eea73 ui: use I/O descriptors internally
Idan Kamara <idankk86@gmail.com>
parents: 14494
diff changeset
117 p.restore()
12703
40bb5853fc4b wireproto: introduce pusherr() to deal with "unsynced changes" error
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11626
diff changeset
118 rsp = '0\n%s\n' % rsp.res
18352
e33b9b92a200 hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents: 18346
diff changeset
119 req.respond(HTTP_OK, HGTYPE, body=rsp)
e33b9b92a200 hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents: 18346
diff changeset
120 return []
15017
f4522df38c65 wireproto: add out-of-band error class to allow remote repo to report errors
Andrew Pritchard <andrewp@fogcreek.com>
parents: 14614
diff changeset
121 elif isinstance(rsp, wireproto.ooberror):
f4522df38c65 wireproto: add out-of-band error class to allow remote repo to report errors
Andrew Pritchard <andrewp@fogcreek.com>
parents: 14614
diff changeset
122 rsp = rsp.message
18352
e33b9b92a200 hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents: 18346
diff changeset
123 req.respond(HTTP_OK, HGERRTYPE, body=rsp)
e33b9b92a200 hgweb: pass the actual response body to request.response, not just the length
Mads Kiilerich <mads@kiilerich.com>
parents: 18346
diff changeset
124 return []