Mercurial > public > mercurial-scm > hg

annotate hgext/lfs/blobstore.py @ 35280:8e72f9152c4d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
lfs: introduce a user level cache for lfs files This is the same mechanism in place for largefiles, and solves several problems working with multiple local repositories. The existing largefiles method is reused in place, because I suspect that there are other functions that can be shared. If we wait a bit to identify more before `hg cp lfutil.py ...`, the history will be easier to trace. The push between repo14 and repo15 in test-lfs.t arguably shouldn't be uploading any files with a local push. Maybe we can revisit that when `hg push` without 'lfs.url' can upload files to the push destination. Then it would be consistent for blobs in a local push to be linked to the local destination's cache. The cache property is added to run-tests.py, the same as the largefiles property, so that test generated files don't pollute the real location. Having files available locally broke a couple existing lfs-test-server tests, so the cache is cleared in a few places to force file download.
author Matt Harbison <matt_harbison@yahoo.com>
date Wed, 06 Dec 2017 22:56:15 -0500
parents 07e97998d385
children 79968f91ad0c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
1 # blobstore.py - local and remote (speaking Git-LFS protocol) blob storages
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
2 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
3 # Copyright 2017 Facebook, Inc.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
4 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
7
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
8 from __future__ import absolute_import
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
9
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
10 import json
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
11 import os
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
12 import re
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
13
35099
b8e5fb8d2389 lfs: quiesce check-module-import warnings
Matt Harbison <matt_harbison@yahoo.com>
parents: 35098
diff changeset
14 from mercurial.i18n import _
b8e5fb8d2389 lfs: quiesce check-module-import warnings
Matt Harbison <matt_harbison@yahoo.com>
parents: 35098
diff changeset
15
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
16 from mercurial import (
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
17 error,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
18 url as urlmod,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
19 util,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
20 vfs as vfsmod,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
21 )
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
22
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
23 from ..largefiles import lfutil
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
24
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
25 # 64 bytes for SHA256
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
26 _lfsre = re.compile(r'\A[a-f0-9]{64}\Z')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
27
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
28 class lfsvfs(vfsmod.vfs):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
29 def join(self, path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
30 """split the path at first two characters, like: XX/XXXXX..."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
31 if not _lfsre.match(path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
32 raise error.ProgrammingError('unexpected lfs path: %s' % path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
33 return super(lfsvfs, self).join(path[0:2], path[2:])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
34
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
35 class filewithprogress(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
36 """a file-like object that supports __len__ and read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
37
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
38 Useful to provide progress information for how many bytes are read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
39 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
40
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
41 def __init__(self, fp, callback):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
42 self._fp = fp
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
43 self._callback = callback # func(readsize)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
44 fp.seek(0, os.SEEK_END)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
45 self._len = fp.tell()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
46 fp.seek(0)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
47
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
48 def __len__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
49 return self._len
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
50
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
51 def read(self, size):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
52 if self._fp is None:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
53 return b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
54 data = self._fp.read(size)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
55 if data:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
56 if self._callback:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
57 self._callback(len(data))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
58 else:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
59 self._fp.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
60 self._fp = None
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
61 return data
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
62
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
63 class local(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
64 """Local blobstore for large file contents.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
65
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
66 This blobstore is used both as a cache and as a staging area for large blobs
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
67 to be uploaded to the remote blobstore.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
68 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
69
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
70 def __init__(self, repo):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
71 fullpath = repo.svfs.join('lfs/objects')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
72 self.vfs = lfsvfs(fullpath)
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
73 usercache = lfutil._usercachedir(repo.ui, 'lfs')
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
74 self.cachevfs = lfsvfs(usercache)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
75
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
76 def write(self, oid, data):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
77 """Write blob to local blobstore."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
78 with self.vfs(oid, 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
79 fp.write(data)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
80
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
81 # XXX: should we verify the content of the cache, and hardlink back to
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
82 # the local store on success, but truncate, write and link on failure?
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
83 if not self.cachevfs.exists(oid):
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
84 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
85
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
86 def read(self, oid):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
87 """Read blob from local blobstore."""
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
88 if not self.vfs.exists(oid):
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
89 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
90 return self.vfs.read(oid)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
91
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
92 def has(self, oid):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
93 """Returns True if the local blobstore contains the requested blob,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
94 False otherwise."""
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
95 return self.cachevfs.exists(oid) or self.vfs.exists(oid)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
96
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
97 class _gitlfsremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
98
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
99 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
100 ui = repo.ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
101 self.ui = ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
102 baseurl, authinfo = url.authinfo()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
103 self.baseurl = baseurl.rstrip('/')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
104 self.urlopener = urlmod.opener(ui, authinfo)
35100
07e97998d385 lfs: register config options
Matt Harbison <matt_harbison@yahoo.com>
parents: 35099
diff changeset
105 self.retry = ui.configint('lfs', 'retry')
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
106
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
107 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
108 """Batch upload from local to remote blobstore."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
109 self._batch(pointers, fromstore, 'upload')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
110
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
111 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
112 """Batch download from remote to local blostore."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
113 self._batch(pointers, tostore, 'download')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
114
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
115 def _batchrequest(self, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
116 """Get metadata about objects pointed by pointers for given action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
117
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
118 Return decoded JSON object like {'objects': [{'oid': '', 'size': 1}]}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
119 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
120 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
121 objects = [{'oid': p.oid(), 'size': p.size()} for p in pointers]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
122 requestdata = json.dumps({
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
123 'objects': objects,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
124 'operation': action,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
125 })
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
126 batchreq = util.urlreq.request('%s/objects/batch' % self.baseurl,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
127 data=requestdata)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
128 batchreq.add_header('Accept', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
129 batchreq.add_header('Content-Type', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
130 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
131 rawjson = self.urlopener.open(batchreq).read()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
132 except util.urlerr.httperror as ex:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
133 raise LfsRemoteError(_('LFS HTTP error: %s (action=%s)')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
134 % (ex, action))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
135 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
136 response = json.loads(rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
137 except ValueError:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
138 raise LfsRemoteError(_('LFS server returns invalid JSON: %s')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
139 % rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
140 return response
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
141
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
142 def _checkforservererror(self, pointers, responses):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
143 """Scans errors from objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
144
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
145 Returns LfsRemoteError if any objects has an error"""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
146 for response in responses:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
147 error = response.get('error')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
148 if error:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
149 ptrmap = {p.oid(): p for p in pointers}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
150 p = ptrmap.get(response['oid'], None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
151 if error['code'] == 404 and p:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
152 filename = getattr(p, 'filename', 'unknown')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
153 raise LfsRemoteError(
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
154 _(('LFS server error. Remote object '
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
155 'for file %s not found: %r')) % (filename, response))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
156 raise LfsRemoteError(_('LFS server error: %r') % response)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
157
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
158 def _extractobjects(self, response, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
159 """extract objects from response of the batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
160
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
161 response: parsed JSON object returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
162 return response['objects'] filtered by action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
163 raise if any object has an error
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
164 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
165 # Scan errors from objects - fail early
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
166 objects = response.get('objects', [])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
167 self._checkforservererror(pointers, objects)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
168
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
169 # Filter objects with given action. Practically, this skips uploading
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
170 # objects which exist in the server.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
171 filteredobjects = [o for o in objects if action in o.get('actions', [])]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
172 # But for downloading, we want all objects. Therefore missing objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
173 # should be considered an error.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
174 if action == 'download':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
175 if len(filteredobjects) < len(objects):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
176 missing = [o.get('oid', '?')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
177 for o in objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
178 if action not in o.get('actions', [])]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
179 raise LfsRemoteError(
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
180 _('LFS server claims required objects do not exist:\n%s')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
181 % '\n'.join(missing))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
182
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
183 return filteredobjects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
184
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
185 def _basictransfer(self, obj, action, localstore, progress=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
186 """Download or upload a single object using basic transfer protocol
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
187
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
188 obj: dict, an object description returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
189 action: string, one of ['upload', 'download']
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
190 localstore: blobstore.local
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
191
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
192 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/\
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
193 basic-transfers.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
194 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
195 oid = str(obj['oid'])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
196
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
197 href = str(obj['actions'][action].get('href'))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
198 headers = obj['actions'][action].get('header', {}).items()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
199
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
200 request = util.urlreq.request(href)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
201 if action == 'upload':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
202 # If uploading blobs, read data from local blobstore.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
203 request.data = filewithprogress(localstore.vfs(oid), progress)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
204 request.get_method = lambda: 'PUT'
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
205
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
206 for k, v in headers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
207 request.add_header(k, v)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
208
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
209 response = b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
210 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
211 req = self.urlopener.open(request)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
212 while True:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
213 data = req.read(1048576)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
214 if not data:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
215 break
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
216 if action == 'download' and progress:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
217 progress(len(data))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
218 response += data
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
219 except util.urlerr.httperror as ex:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
220 raise LfsRemoteError(_('HTTP error: %s (oid=%s, action=%s)')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
221 % (ex, oid, action))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
222
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
223 if action == 'download':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
224 # If downloading blobs, store downloaded data to local blobstore
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
225 localstore.write(oid, response)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
226
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
227 def _batch(self, pointers, localstore, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
228 if action not in ['upload', 'download']:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
229 raise error.ProgrammingError('invalid Git-LFS action: %s' % action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
230
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
231 response = self._batchrequest(pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
232 prunningsize = [0]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
233 objects = self._extractobjects(response, pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
234 total = sum(x.get('size', 0) for x in objects)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
235 topic = {'upload': _('lfs uploading'),
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
236 'download': _('lfs downloading')}[action]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
237 if self.ui.verbose and len(objects) > 1:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
238 self.ui.write(_('lfs: need to transfer %d objects (%s)\n')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
239 % (len(objects), util.bytecount(total)))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
240 self.ui.progress(topic, 0, total=total)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
241 def progress(size):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
242 # advance progress bar by "size" bytes
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
243 prunningsize[0] += size
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
244 self.ui.progress(topic, prunningsize[0], total=total)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
245 for obj in sorted(objects, key=lambda o: o.get('oid')):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
246 objsize = obj.get('size', 0)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
247 if self.ui.verbose:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
248 if action == 'download':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
249 msg = _('lfs: downloading %s (%s)\n')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
250 elif action == 'upload':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
251 msg = _('lfs: uploading %s (%s)\n')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
252 self.ui.write(msg % (obj.get('oid'), util.bytecount(objsize)))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
253 origrunningsize = prunningsize[0]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
254 retry = self.retry
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
255 while True:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
256 prunningsize[0] = origrunningsize
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
257 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
258 self._basictransfer(obj, action, localstore,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
259 progress=progress)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
260 break
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
261 except Exception as ex:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
262 if retry > 0:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
263 if self.ui.verbose:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
264 self.ui.write(
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
265 _('lfs: failed: %r (remaining retry %d)\n')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
266 % (ex, retry))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
267 retry -= 1
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
268 continue
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
269 raise
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
270
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
271 self.ui.progress(topic, pos=None, total=total)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
272
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
273 def __del__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
274 # copied from mercurial/httppeer.py
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
275 urlopener = getattr(self, 'urlopener', None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
276 if urlopener:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
277 for h in urlopener.handlers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
278 h.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
279 getattr(h, "close_all", lambda : None)()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
280
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
281 class _dummyremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
282 """Dummy store storing blobs to temp directory."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
283
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
284 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
285 fullpath = repo.vfs.join('lfs', url.path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
286 self.vfs = lfsvfs(fullpath)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
287
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
288 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
289 for p in pointers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
290 content = fromstore.read(p.oid())
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
291 with self.vfs(p.oid(), 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
292 fp.write(content)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
293
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
294 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
295 for p in pointers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
296 content = self.vfs.read(p.oid())
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
297 tostore.write(p.oid(), content)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
298
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
299 class _nullremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
300 """Null store storing blobs to /dev/null."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
301
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
302 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
303 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
304
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
305 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
306 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
307
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
308 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
309 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
310
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
311 class _promptremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
312 """Prompt user to set lfs.url when accessed."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
313
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
314 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
315 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
316
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
317 def writebatch(self, pointers, fromstore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
318 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
319
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
320 def readbatch(self, pointers, tostore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
321 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
322
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
323 def _prompt(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
324 raise error.Abort(_('lfs.url needs to be configured'))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
325
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
326 _storemap = {
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
327 'https': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
328 'http': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
329 'file': _dummyremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
330 'null': _nullremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
331 None: _promptremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
332 }
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
333
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
334 def remote(repo):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
335 """remotestore factory. return a store in _storemap depending on config"""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
336 defaulturl = ''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
337
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
338 # convert deprecated configs to the new url. TODO: remove this if other
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
339 # places are migrated to the new url config.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
340 # deprecated config: lfs.remotestore
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
341 deprecatedstore = repo.ui.config('lfs', 'remotestore')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
342 if deprecatedstore == 'dummy':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
343 # deprecated config: lfs.remotepath
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
344 defaulturl = 'file://' + repo.ui.config('lfs', 'remotepath')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
345 elif deprecatedstore == 'git-lfs':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
346 # deprecated config: lfs.remoteurl
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
347 defaulturl = repo.ui.config('lfs', 'remoteurl')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
348 elif deprecatedstore == 'null':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
349 defaulturl = 'null://'
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
350
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
351 url = util.url(repo.ui.config('lfs', 'url', defaulturl))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
352 scheme = url.scheme
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
353 if scheme not in _storemap:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
354 raise error.Abort(_('lfs: unknown url scheme: %s') % scheme)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
355 return _storemap[scheme](repo, url)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
356
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
357 class LfsRemoteError(error.RevlogError):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
358 pass