Mercurial > public > mercurial-scm > hg

annotate hgext/lfs/wireprotolfsserver.py @ 37148:ea6fc58524d7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
lfs: add server side support for the Batch API
author Matt Harbison <matt_harbison@yahoo.com>
date Sat, 17 Mar 2018 01:47:57 -0400
parents a2566597acb5
children cc0a6ea95d98
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
1 # wireprotolfsserver.py - lfs protocol server side implementation
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
2 #
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
3 # Copyright 2018 Matt Harbison <matt_harbison@yahoo.com>
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
4 #
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
7
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
8 from __future__ import absolute_import
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
9
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
10 import datetime
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
11 import errno
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
12 import json
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
13
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
14 from mercurial.hgweb import (
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
15 common as hgwebcommon,
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
16 )
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
17
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
18 from mercurial import (
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
19 pycompat,
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
20 )
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
21
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
22 HTTP_OK = hgwebcommon.HTTP_OK
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
23 HTTP_BAD_REQUEST = hgwebcommon.HTTP_BAD_REQUEST
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
24
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
25 def handlewsgirequest(orig, rctx, req, res, checkperm):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
26 """Wrap wireprotoserver.handlewsgirequest() to possibly process an LFS
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
27 request if it is left unprocessed by the wrapped method.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
28 """
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
29 if orig(rctx, req, res, checkperm):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
30 return True
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
31
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
32 if not req.dispatchpath:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
33 return False
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
34
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
35 try:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
36 if req.dispatchpath == b'.git/info/lfs/objects/batch':
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
37 checkperm(rctx, req, 'pull')
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
38 return _processbatchrequest(rctx.repo, req, res)
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
39 # TODO: reserve and use a path in the proposed http wireprotocol /api/
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
40 # namespace?
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
41 elif req.dispatchpath.startswith(b'.hg/lfs/objects'):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
42 return _processbasictransfer(rctx.repo, req, res,
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
43 lambda perm:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
44 checkperm(rctx, req, perm))
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
45 return False
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
46 except hgwebcommon.ErrorResponse as e:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
47 # XXX: copied from the handler surrounding wireprotoserver._callhttp()
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
48 # in the wrapped function. Should this be moved back to hgweb to
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
49 # be a common handler?
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
50 for k, v in e.headers:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
51 res.headers[k] = v
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
52 res.status = hgwebcommon.statusmessage(e.code, pycompat.bytestr(e))
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
53 res.setbodybytes(b'0\n%s\n' % pycompat.bytestr(e))
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
54 return True
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
55
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
56 def _sethttperror(res, code, message=None):
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
57 res.status = hgwebcommon.statusmessage(code, message=message)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
58 res.headers[b'Content-Type'] = b'text/plain; charset=utf-8'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
59 res.setbodybytes(b'')
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
60
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
61 def _processbatchrequest(repo, req, res):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
62 """Handle a request for the Batch API, which is the gateway to granting file
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
63 access.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
64
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
65 https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
66 """
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
67
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
68 # Mercurial client request:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
69 #
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
70 # HOST: localhost:$HGPORT
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
71 # ACCEPT: application/vnd.git-lfs+json
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
72 # ACCEPT-ENCODING: identity
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
73 # USER-AGENT: git-lfs/2.3.4 (Mercurial 4.5.2+1114-f48b9754f04c+20180316)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
74 # Content-Length: 125
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
75 # Content-Type: application/vnd.git-lfs+json
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
76 #
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
77 # {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
78 # "objects": [
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
79 # {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
80 # "oid": "31cf...8e5b"
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
81 # "size": 12
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
82 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
83 # ]
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
84 # "operation": "upload"
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
85 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
86
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
87 if (req.method != b'POST'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
88 or req.headers[b'Content-Type'] != b'application/vnd.git-lfs+json'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
89 or req.headers[b'Accept'] != b'application/vnd.git-lfs+json'):
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
90 # TODO: figure out what the proper handling for a bad request to the
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
91 # Batch API is.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
92 _sethttperror(res, HTTP_BAD_REQUEST, b'Invalid Batch API request')
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
93 return True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
94
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
95 # XXX: specify an encoding?
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
96 lfsreq = json.loads(req.bodyfh.read())
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
97
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
98 # If no transfer handlers are explicitly requested, 'basic' is assumed.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
99 if 'basic' not in lfsreq.get('transfers', ['basic']):
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
100 _sethttperror(res, HTTP_BAD_REQUEST,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
101 b'Only the basic LFS transfer handler is supported')
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
102 return True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
103
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
104 operation = lfsreq.get('operation')
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
105 if operation not in ('upload', 'download'):
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
106 _sethttperror(res, HTTP_BAD_REQUEST,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
107 b'Unsupported LFS transfer operation: %s' % operation)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
108 return True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
109
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
110 localstore = repo.svfs.lfslocalblobstore
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
111
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
112 objects = [p for p in _batchresponseobjects(req, lfsreq.get('objects', []),
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
113 operation, localstore)]
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
114
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
115 rsp = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
116 'transfer': 'basic',
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
117 'objects': objects,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
118 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
119
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
120 res.status = hgwebcommon.statusmessage(HTTP_OK)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
121 res.headers[b'Content-Type'] = b'application/vnd.git-lfs+json'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
122 res.setbodybytes(pycompat.bytestr(json.dumps(rsp)))
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
123
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
124 return True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
125
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
126 def _batchresponseobjects(req, objects, action, store):
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
127 """Yield one dictionary of attributes for the Batch API response for each
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
128 object in the list.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
129
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
130 req: The parsedrequest for the Batch API request
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
131 objects: The list of objects in the Batch API object request list
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
132 action: 'upload' or 'download'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
133 store: The local blob store for servicing requests"""
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
134
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
135 # Successful lfs-test-server response to solict an upload:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
136 # {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
137 # u'objects': [{
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
138 # u'size': 12,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
139 # u'oid': u'31cf...8e5b',
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
140 # u'actions': {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
141 # u'upload': {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
142 # u'href': u'http://localhost:$HGPORT/objects/31cf...8e5b',
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
143 # u'expires_at': u'0001-01-01T00:00:00Z',
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
144 # u'header': {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
145 # u'Accept': u'application/vnd.git-lfs'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
146 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
147 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
148 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
149 # }]
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
150 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
151
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
152 # TODO: Sort out the expires_at/expires_in/authenticated keys.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
153
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
154 for obj in objects:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
155 # Convert unicode to ASCII to create a filesystem path
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
156 oid = obj.get('oid').encode('ascii')
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
157 rsp = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
158 'oid': oid,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
159 'size': obj.get('size'), # XXX: should this check the local size?
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
160 #'authenticated': True,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
161 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
162
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
163 exists = True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
164 verifies = False
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
165
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
166 # Verify an existing file on the upload request, so that the client is
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
167 # solicited to re-upload if it corrupt locally. Download requests are
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
168 # also verified, so the error can be flagged in the Batch API response.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
169 # (Maybe we can use this to short circuit the download for `hg verify`,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
170 # IFF the client can assert that the remote end is an hg server.)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
171 # Otherwise, it's potentially overkill on download, since it is also
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
172 # verified as the file is streamed to the caller.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
173 try:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
174 verifies = store.verify(oid)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
175 except IOError as inst:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
176 if inst.errno != errno.ENOENT:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
177 rsp['error'] = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
178 'code': 500,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
179 'message': inst.strerror or 'Internal Server Server'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
180 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
181 yield rsp
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
182 continue
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
183
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
184 exists = False
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
185
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
186 # Items are always listed for downloads. They are dropped for uploads
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
187 # IFF they already exist locally.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
188 if action == 'download':
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
189 if not exists:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
190 rsp['error'] = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
191 'code': 404,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
192 'message': "The object does not exist"
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
193 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
194 yield rsp
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
195 continue
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
196
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
197 elif not verifies:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
198 rsp['error'] = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
199 'code': 422, # XXX: is this the right code?
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
200 'message': "The object is corrupt"
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
201 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
202 yield rsp
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
203 continue
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
204
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
205 elif verifies:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
206 yield rsp # Skip 'actions': already uploaded
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
207 continue
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
208
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
209 expiresat = datetime.datetime.now() + datetime.timedelta(minutes=10)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
210
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
211 rsp['actions'] = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
212 '%s' % action: {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
213 # TODO: Account for the --prefix, if any.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
214 'href': '%s/.hg/lfs/objects/%s' % (req.baseurl, oid),
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
215 # datetime.isoformat() doesn't include the 'Z' suffix
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
216 "expires_at": expiresat.strftime('%Y-%m-%dT%H:%M:%SZ'),
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
217 'header': {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
218 # The spec doesn't mention the Accept header here, but avoid
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
219 # a gratuitous deviation from lfs-test-server in the test
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
220 # output.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
221 'Accept': 'application/vnd.git-lfs'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
222 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
223 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
224 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
225
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
226 yield rsp
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
227
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
228 def _processbasictransfer(repo, req, res, checkperm):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
229 """Handle a single file upload (PUT) or download (GET) action for the Basic
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
230 Transfer Adapter.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
231
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
232 After determining if the request is for an upload or download, the access
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
233 must be checked by calling ``checkperm()`` with either 'pull' or 'upload'
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
234 before accessing the files.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
235
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
236 https://github.com/git-lfs/git-lfs/blob/master/docs/api/basic-transfers.md
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
237 """
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
238
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
239 method = req.method
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
240
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
241 if method == b'PUT':
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
242 checkperm('upload')
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
243 elif method == b'GET':
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
244 checkperm('pull')
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
245
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
246 return False