annotate hgext/lfs/wireprotolfsserver.py @ 39457:a913d2892e17

lfs: ensure the blob is linked to the remote store on skipped uploads I noticed a "missing" blob when pushing two repositories with common blobs to a fresh server, and then running `hg verify` as a user different from the one running the web server. When pushing the second repo, several of the blobs already existed in the user cache, so the server indicated to the client that it doesn't need to upload the blobs. That's good enough for the web server process to serve up in the future. But a different user has a different cache by default, so verify complains that `lfs.url` needs to be set, because it wants to fetch the missing blobs. Aside from that corner case, it's better to keep all of the blobs in the repo whenever possible. Especially since the largefiles wiki says the user cache can be deleted at any time to reclaim disk space- users switching over may have the same expectations.
author Matt Harbison <matt_harbison@yahoo.com>
date Thu, 06 Sep 2018 00:51:21 -0400
parents 3790efb388ca
children 84d61fdcefa5
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
1 # wireprotolfsserver.py - lfs protocol server side implementation
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
2 #
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
3 # Copyright 2018 Matt Harbison <matt_harbison@yahoo.com>
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
4 #
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
7
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
8 from __future__ import absolute_import
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
9
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
10 import datetime
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
11 import errno
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
12 import json
37690
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
13 import traceback
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
14
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
15 from mercurial.hgweb import (
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
16 common as hgwebcommon,
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
17 )
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
18
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
19 from mercurial import (
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
20 pycompat,
38178
3790efb388ca lfs: bypass wrapped functions when reposetup() hasn't been called (issue5902)
Matt Harbison <matt_harbison@yahoo.com>
parents: 37766
diff changeset
21 util,
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
22 )
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
23
37692
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
24 from . import blobstore
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
25
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
26 HTTP_OK = hgwebcommon.HTTP_OK
37149
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
27 HTTP_CREATED = hgwebcommon.HTTP_CREATED
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
28 HTTP_BAD_REQUEST = hgwebcommon.HTTP_BAD_REQUEST
37250
9640ccf44ac0 lfs: ensure the transfer request is for a known URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 37249
diff changeset
29 HTTP_NOT_FOUND = hgwebcommon.HTTP_NOT_FOUND
37693
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
30 HTTP_METHOD_NOT_ALLOWED = hgwebcommon.HTTP_METHOD_NOT_ALLOWED
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
31 HTTP_NOT_ACCEPTABLE = hgwebcommon.HTTP_NOT_ACCEPTABLE
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
32 HTTP_UNSUPPORTED_MEDIA_TYPE = hgwebcommon.HTTP_UNSUPPORTED_MEDIA_TYPE
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
33
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
34 def handlewsgirequest(orig, rctx, req, res, checkperm):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
35 """Wrap wireprotoserver.handlewsgirequest() to possibly process an LFS
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
36 request if it is left unprocessed by the wrapped method.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
37 """
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
38 if orig(rctx, req, res, checkperm):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
39 return True
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
40
37248
dfb38c4850a9 lfs: add an experimental knob to disable blob serving
Matt Harbison <matt_harbison@yahoo.com>
parents: 37149
diff changeset
41 if not rctx.repo.ui.configbool('experimental', 'lfs.serve'):
dfb38c4850a9 lfs: add an experimental knob to disable blob serving
Matt Harbison <matt_harbison@yahoo.com>
parents: 37149
diff changeset
42 return False
dfb38c4850a9 lfs: add an experimental knob to disable blob serving
Matt Harbison <matt_harbison@yahoo.com>
parents: 37149
diff changeset
43
38178
3790efb388ca lfs: bypass wrapped functions when reposetup() hasn't been called (issue5902)
Matt Harbison <matt_harbison@yahoo.com>
parents: 37766
diff changeset
44 if not util.safehasattr(rctx.repo.svfs, 'lfslocalblobstore'):
3790efb388ca lfs: bypass wrapped functions when reposetup() hasn't been called (issue5902)
Matt Harbison <matt_harbison@yahoo.com>
parents: 37766
diff changeset
45 return False
3790efb388ca lfs: bypass wrapped functions when reposetup() hasn't been called (issue5902)
Matt Harbison <matt_harbison@yahoo.com>
parents: 37766
diff changeset
46
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
47 if not req.dispatchpath:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
48 return False
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
49
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
50 try:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
51 if req.dispatchpath == b'.git/info/lfs/objects/batch':
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
52 checkperm(rctx, req, 'pull')
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
53 return _processbatchrequest(rctx.repo, req, res)
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
54 # TODO: reserve and use a path in the proposed http wireprotocol /api/
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
55 # namespace?
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
56 elif req.dispatchpath.startswith(b'.hg/lfs/objects'):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
57 return _processbasictransfer(rctx.repo, req, res,
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
58 lambda perm:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
59 checkperm(rctx, req, perm))
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
60 return False
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
61 except hgwebcommon.ErrorResponse as e:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
62 # XXX: copied from the handler surrounding wireprotoserver._callhttp()
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
63 # in the wrapped function. Should this be moved back to hgweb to
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
64 # be a common handler?
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
65 for k, v in e.headers:
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
66 res.headers[k] = v
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
67 res.status = hgwebcommon.statusmessage(e.code, pycompat.bytestr(e))
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
68 res.setbodybytes(b'0\n%s\n' % pycompat.bytestr(e))
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
69 return True
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
70
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
71 def _sethttperror(res, code, message=None):
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
72 res.status = hgwebcommon.statusmessage(code, message=message)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
73 res.headers[b'Content-Type'] = b'text/plain; charset=utf-8'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
74 res.setbodybytes(b'')
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
75
37690
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
76 def _logexception(req):
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
77 """Write information about the current exception to wsgi.errors."""
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
78 tb = pycompat.sysbytes(traceback.format_exc())
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
79 errorlog = req.rawenv[r'wsgi.errors']
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
80
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
81 uri = b''
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
82 if req.apppath:
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
83 uri += req.apppath
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
84 uri += b'/' + req.dispatchpath
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
85
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
86 errorlog.write(b"Exception happened while processing request '%s':\n%s" %
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
87 (uri, tb))
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
88
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
89 def _processbatchrequest(repo, req, res):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
90 """Handle a request for the Batch API, which is the gateway to granting file
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
91 access.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
92
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
93 https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
94 """
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
95
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
96 # Mercurial client request:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
97 #
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
98 # HOST: localhost:$HGPORT
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
99 # ACCEPT: application/vnd.git-lfs+json
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
100 # ACCEPT-ENCODING: identity
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
101 # USER-AGENT: git-lfs/2.3.4 (Mercurial 4.5.2+1114-f48b9754f04c+20180316)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
102 # Content-Length: 125
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
103 # Content-Type: application/vnd.git-lfs+json
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
104 #
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
105 # {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
106 # "objects": [
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
107 # {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
108 # "oid": "31cf...8e5b"
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
109 # "size": 12
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
110 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
111 # ]
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
112 # "operation": "upload"
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
113 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
114
37693
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
115 if req.method != b'POST':
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
116 _sethttperror(res, HTTP_METHOD_NOT_ALLOWED)
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
117 return True
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
118
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
119 if req.headers[b'Content-Type'] != b'application/vnd.git-lfs+json':
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
120 _sethttperror(res, HTTP_UNSUPPORTED_MEDIA_TYPE)
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
121 return True
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
122
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
123 if req.headers[b'Accept'] != b'application/vnd.git-lfs+json':
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
124 _sethttperror(res, HTTP_NOT_ACCEPTABLE)
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
125 return True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
126
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
127 # XXX: specify an encoding?
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
128 lfsreq = json.loads(req.bodyfh.read())
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
129
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
130 # If no transfer handlers are explicitly requested, 'basic' is assumed.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
131 if 'basic' not in lfsreq.get('transfers', ['basic']):
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
132 _sethttperror(res, HTTP_BAD_REQUEST,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
133 b'Only the basic LFS transfer handler is supported')
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
134 return True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
135
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
136 operation = lfsreq.get('operation')
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
137 if operation not in ('upload', 'download'):
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
138 _sethttperror(res, HTTP_BAD_REQUEST,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
139 b'Unsupported LFS transfer operation: %s' % operation)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
140 return True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
141
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
142 localstore = repo.svfs.lfslocalblobstore
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
143
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
144 objects = [p for p in _batchresponseobjects(req, lfsreq.get('objects', []),
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
145 operation, localstore)]
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
146
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
147 rsp = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
148 'transfer': 'basic',
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
149 'objects': objects,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
150 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
151
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
152 res.status = hgwebcommon.statusmessage(HTTP_OK)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
153 res.headers[b'Content-Type'] = b'application/vnd.git-lfs+json'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
154 res.setbodybytes(pycompat.bytestr(json.dumps(rsp)))
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
155
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
156 return True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
157
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
158 def _batchresponseobjects(req, objects, action, store):
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
159 """Yield one dictionary of attributes for the Batch API response for each
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
160 object in the list.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
161
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
162 req: The parsedrequest for the Batch API request
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
163 objects: The list of objects in the Batch API object request list
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
164 action: 'upload' or 'download'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
165 store: The local blob store for servicing requests"""
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
166
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
167 # Successful lfs-test-server response to solict an upload:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
168 # {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
169 # u'objects': [{
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
170 # u'size': 12,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
171 # u'oid': u'31cf...8e5b',
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
172 # u'actions': {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
173 # u'upload': {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
174 # u'href': u'http://localhost:$HGPORT/objects/31cf...8e5b',
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
175 # u'expires_at': u'0001-01-01T00:00:00Z',
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
176 # u'header': {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
177 # u'Accept': u'application/vnd.git-lfs'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
178 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
179 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
180 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
181 # }]
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
182 # }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
183
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
184 # TODO: Sort out the expires_at/expires_in/authenticated keys.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
185
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
186 for obj in objects:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
187 # Convert unicode to ASCII to create a filesystem path
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
188 oid = obj.get('oid').encode('ascii')
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
189 rsp = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
190 'oid': oid,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
191 'size': obj.get('size'), # XXX: should this check the local size?
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
192 #'authenticated': True,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
193 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
194
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
195 exists = True
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
196 verifies = False
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
197
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
198 # Verify an existing file on the upload request, so that the client is
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
199 # solicited to re-upload if it corrupt locally. Download requests are
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
200 # also verified, so the error can be flagged in the Batch API response.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
201 # (Maybe we can use this to short circuit the download for `hg verify`,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
202 # IFF the client can assert that the remote end is an hg server.)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
203 # Otherwise, it's potentially overkill on download, since it is also
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
204 # verified as the file is streamed to the caller.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
205 try:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
206 verifies = store.verify(oid)
39457
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 38178
diff changeset
207 if verifies and action == 'upload':
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 38178
diff changeset
208 # The client will skip this upload, but make sure it remains
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 38178
diff changeset
209 # available locally.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 38178
diff changeset
210 store.linkfromusercache(oid)
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
211 except IOError as inst:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
212 if inst.errno != errno.ENOENT:
37690
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
213 _logexception(req)
726c4102db9e lfs: log information about Internal Server Errors reported in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37617
diff changeset
214
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
215 rsp['error'] = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
216 'code': 500,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
217 'message': inst.strerror or 'Internal Server Server'
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
218 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
219 yield rsp
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
220 continue
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
221
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
222 exists = False
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
223
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
224 # Items are always listed for downloads. They are dropped for uploads
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
225 # IFF they already exist locally.
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
226 if action == 'download':
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
227 if not exists:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
228 rsp['error'] = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
229 'code': 404,
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
230 'message': "The object does not exist"
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
231 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
232 yield rsp
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
233 continue
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
234
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
235 elif not verifies:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
236 rsp['error'] = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
237 'code': 422, # XXX: is this the right code?
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
238 'message': "The object is corrupt"
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
239 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
240 yield rsp
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
241 continue
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
242
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
243 elif verifies:
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
244 yield rsp # Skip 'actions': already uploaded
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
245 continue
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
246
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
247 expiresat = datetime.datetime.now() + datetime.timedelta(minutes=10)
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
248
37766
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
249 def _buildheader():
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
250 # The spec doesn't mention the Accept header here, but avoid
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
251 # a gratuitous deviation from lfs-test-server in the test
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
252 # output.
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
253 hdr = {
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
254 'Accept': 'application/vnd.git-lfs'
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
255 }
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
256
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
257 auth = req.headers.get('Authorization', '')
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
258 if auth.startswith('Basic '):
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
259 hdr['Authorization'] = auth
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
260
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
261 return hdr
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
262
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
263 rsp['actions'] = {
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
264 '%s' % action: {
37617
b03f2e0fdb88 lfs: teach the blob server to handle --prefix
Matt Harbison <matt_harbison@yahoo.com>
parents: 37250
diff changeset
265 'href': '%s%s/.hg/lfs/objects/%s'
b03f2e0fdb88 lfs: teach the blob server to handle --prefix
Matt Harbison <matt_harbison@yahoo.com>
parents: 37250
diff changeset
266 % (req.baseurl, req.apppath, oid),
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
267 # datetime.isoformat() doesn't include the 'Z' suffix
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
268 "expires_at": expiresat.strftime('%Y-%m-%dT%H:%M:%SZ'),
37766
925707ac2855 lfs: add the 'Authorization' property to the Batch API response, if present
Matt Harbison <matt_harbison@yahoo.com>
parents: 37693
diff changeset
269 'header': _buildheader(),
37148
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
270 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
271 }
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
272
ea6fc58524d7 lfs: add server side support for the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 37147
diff changeset
273 yield rsp
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
274
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
275 def _processbasictransfer(repo, req, res, checkperm):
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
276 """Handle a single file upload (PUT) or download (GET) action for the Basic
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
277 Transfer Adapter.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
278
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
279 After determining if the request is for an upload or download, the access
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
280 must be checked by calling ``checkperm()`` with either 'pull' or 'upload'
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
281 before accessing the files.
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
282
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
283 https://github.com/git-lfs/git-lfs/blob/master/docs/api/basic-transfers.md
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
284 """
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
285
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
286 method = req.method
37249
fe061e47a2cf lfs: avoid an improper usage of os.path.basename() to parse a URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 37248
diff changeset
287 oid = req.dispatchparts[-1]
37149
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
288 localstore = repo.svfs.lfslocalblobstore
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
289
37250
9640ccf44ac0 lfs: ensure the transfer request is for a known URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 37249
diff changeset
290 if len(req.dispatchparts) != 4:
9640ccf44ac0 lfs: ensure the transfer request is for a known URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 37249
diff changeset
291 _sethttperror(res, HTTP_NOT_FOUND)
9640ccf44ac0 lfs: ensure the transfer request is for a known URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 37249
diff changeset
292 return True
9640ccf44ac0 lfs: ensure the transfer request is for a known URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 37249
diff changeset
293
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
294 if method == b'PUT':
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
295 checkperm('upload')
37149
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
296
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
297 # TODO: verify Content-Type?
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
298
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
299 existed = localstore.has(oid)
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
300
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
301 # TODO: how to handle timeouts? The body proxy handles limiting to
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
302 # Content-Length, but what happens if a client sends less than it
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
303 # says it will?
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
304
37692
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
305 statusmessage = hgwebcommon.statusmessage
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
306 try:
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
307 localstore.download(oid, req.bodyfh)
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
308 res.status = statusmessage(HTTP_OK if existed else HTTP_CREATED)
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
309 except blobstore.LfsCorruptionError:
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
310 _logexception(req)
37149
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
311
37692
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
312 # XXX: Is this the right code?
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
313 res.status = statusmessage(422, b'corrupt blob')
37149
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
314
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
315 # There's no payload here, but this is the header that lfs-test-server
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
316 # sends back. This eliminates some gratuitous test output conditionals.
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
317 res.headers[b'Content-Type'] = b'text/plain; charset=utf-8'
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
318 res.setbodybytes(b'')
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
319
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
320 return True
37147
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
321 elif method == b'GET':
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
322 checkperm('pull')
a2566597acb5 lfs: add basic routing for the server side wire protocol processing
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
323
37149
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
324 res.status = hgwebcommon.statusmessage(HTTP_OK)
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
325 res.headers[b'Content-Type'] = b'application/octet-stream'
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
326
37692
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
327 try:
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
328 # TODO: figure out how to send back the file in chunks, instead of
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
329 # reading the whole thing. (Also figure out how to send back
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
330 # an error status if an IOError occurs after a partial write
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
331 # in that case. Here, everything is read before starting.)
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
332 res.setbodybytes(localstore.read(oid))
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
333 except blobstore.LfsCorruptionError:
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
334 _logexception(req)
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
335
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
336 # XXX: Is this the right code?
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
337 res.status = hgwebcommon.statusmessage(422, b'corrupt blob')
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37690
diff changeset
338 res.setbodybytes(b'')
37149
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
339
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
340 return True
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
341 else:
37693
31a0d47d69b3 lfs: update the HTTP status codes in error cases
Matt Harbison <matt_harbison@yahoo.com>
parents: 37692
diff changeset
342 _sethttperror(res, HTTP_METHOD_NOT_ALLOWED,
37149
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
343 message=b'Unsupported LFS transfer method: %s' % method)
cc0a6ea95d98 lfs: add support for serving blob files
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
344 return True