Mercurial > public > mercurial-scm > hg-stable

annotate hgext/lfs/blobstore.py @ 37149:c37c47e47a95

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
test-lfs: drop trailing ', ' item separators from debug JSON output The trailing space looks weird when conditionalizing the line. The commas shouldn't be necessary because of the indenting. The `lfs-test-server` isn't sending all of the same items (notably, the "transfer" attribute is missing), so having the commas means more lines need to be conditionalized.
author Matt Harbison <matt_harbison@yahoo.com>
date Wed, 14 Mar 2018 23:34:08 -0400
parents 56c7cd067477
children b00bd974eef5
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
1 # blobstore.py - local and remote (speaking Git-LFS protocol) blob storages
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
2 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
3 # Copyright 2017 Facebook, Inc.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
4 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
7
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
8 from __future__ import absolute_import
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
9
35480
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
10 import hashlib
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
11 import json
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
12 import os
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
13 import re
35479
b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry
Matt Harbison <matt_harbison@yahoo.com>
parents: 35477
diff changeset
14 import socket
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
15
35122
b8e5fb8d2389 lfs: quiesce check-module-import warnings
Matt Harbison <matt_harbison@yahoo.com>
parents: 35121
diff changeset
16 from mercurial.i18n import _
b8e5fb8d2389 lfs: quiesce check-module-import warnings
Matt Harbison <matt_harbison@yahoo.com>
parents: 35121
diff changeset
17
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
18 from mercurial import (
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
19 error,
35371
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
20 pathutil,
36630
4da09b46451e lfs: add some bytestring wrappers in blobstore.py
Augie Fackler <augie@google.com>
parents: 36485
diff changeset
21 pycompat,
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
22 url as urlmod,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
23 util,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
24 vfs as vfsmod,
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
25 worker,
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
26 )
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
27
35289
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
28 from ..largefiles import lfutil
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
29
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
30 # 64 bytes for SHA256
36485
9e3cb58c7ab3 py3: make sure regexes are bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35967
diff changeset
31 _lfsre = re.compile(br'\A[a-f0-9]{64}\Z')
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
32
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
33 class lfsvfs(vfsmod.vfs):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
34 def join(self, path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
35 """split the path at first two characters, like: XX/XXXXX..."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
36 if not _lfsre.match(path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
37 raise error.ProgrammingError('unexpected lfs path: %s' % path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
38 return super(lfsvfs, self).join(path[0:2], path[2:])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
39
35371
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
40 def walk(self, path=None, onerror=None):
35403
c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk()
Matt Harbison <matt_harbison@yahoo.com>
parents: 35371
diff changeset
41 """Yield (dirpath, [], oids) tuple for blobs under path
35371
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
42
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
43 Oids only exist in the root of this vfs, so dirpath is always ''.
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
44 """
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
45 root = os.path.normpath(self.base)
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
46 # when dirpath == root, dirpath[prefixlen:] becomes empty
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
47 # because len(dirpath) < prefixlen.
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
48 prefixlen = len(pathutil.normasprefix(root))
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
49 oids = []
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
50
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
51 for dirpath, dirs, files in os.walk(self.reljoin(self.base, path or ''),
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
52 onerror=onerror):
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
53 dirpath = dirpath[prefixlen:]
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
54
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
55 # Silently skip unexpected files and directories
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
56 if len(dirpath) == 2:
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
57 oids.extend([dirpath + f for f in files
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
58 if _lfsre.match(dirpath + f)])
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
59
35403
c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk()
Matt Harbison <matt_harbison@yahoo.com>
parents: 35371
diff changeset
60 yield ('', [], oids)
35371
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35289
diff changeset
61
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
62 class filewithprogress(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
63 """a file-like object that supports __len__ and read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
64
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
65 Useful to provide progress information for how many bytes are read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
66 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
67
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
68 def __init__(self, fp, callback):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
69 self._fp = fp
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
70 self._callback = callback # func(readsize)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
71 fp.seek(0, os.SEEK_END)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
72 self._len = fp.tell()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
73 fp.seek(0)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
74
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
75 def __len__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
76 return self._len
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
77
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
78 def read(self, size):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
79 if self._fp is None:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
80 return b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
81 data = self._fp.read(size)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
82 if data:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
83 if self._callback:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
84 self._callback(len(data))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
85 else:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
86 self._fp.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
87 self._fp = None
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
88 return data
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
89
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
90 class local(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
91 """Local blobstore for large file contents.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
92
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
93 This blobstore is used both as a cache and as a staging area for large blobs
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
94 to be uploaded to the remote blobstore.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
95 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
96
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
97 def __init__(self, repo):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
98 fullpath = repo.svfs.join('lfs/objects')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
99 self.vfs = lfsvfs(fullpath)
35289
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
100 usercache = lfutil._usercachedir(repo.ui, 'lfs')
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
101 self.cachevfs = lfsvfs(usercache)
35477
02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35444
diff changeset
102 self.ui = repo.ui
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
103
35529
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35482
diff changeset
104 def open(self, oid):
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35482
diff changeset
105 """Open a read-only file descriptor to the named blob, in either the
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35482
diff changeset
106 usercache or the local store."""
35541
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35530
diff changeset
107 # The usercache is the most likely place to hold the file. Commit will
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35530
diff changeset
108 # write to both it and the local store, as will anything that downloads
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35530
diff changeset
109 # the blobs. However, things like clone without an update won't
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35530
diff changeset
110 # populate the local store. For an init + push of a local clone,
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35530
diff changeset
111 # the usercache is the only place it _could_ be. If not present, the
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35530
diff changeset
112 # missing file msg here will indicate the local repo, not the usercache.
35529
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35482
diff changeset
113 if self.cachevfs.exists(oid):
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35482
diff changeset
114 return self.cachevfs(oid, 'rb')
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35482
diff changeset
115
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35482
diff changeset
116 return self.vfs(oid, 'rb')
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35482
diff changeset
117
35551
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
118 def download(self, oid, src):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
119 """Read the blob from the remote source in chunks, verify the content,
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
120 and write to this local blobstore."""
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
121 sha256 = hashlib.sha256()
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
122
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
123 with self.vfs(oid, 'wb', atomictemp=True) as fp:
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
124 for chunk in util.filechunkiter(src, size=1048576):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
125 fp.write(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
126 sha256.update(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
127
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
128 realoid = sha256.hexdigest()
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
129 if realoid != oid:
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
130 raise error.Abort(_('corrupt remote lfs object: %s') % oid)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
131
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
132 # XXX: should we verify the content of the cache, and hardlink back to
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
133 # the local store on success, but truncate, write and link on failure?
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
134 if not self.cachevfs.exists(oid):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
135 self.ui.note(_('lfs: adding %s to the usercache\n') % oid)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
136 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35541
diff changeset
137
35553
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
138 def write(self, oid, data):
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
139 """Write blob to local blobstore.
35480
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
140
35553
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
141 This should only be called from the filelog during a commit or similar.
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
142 As such, there is no need to verify the data. Imports from a remote
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
143 store must use ``download()`` instead."""
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
144 with self.vfs(oid, 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
145 fp.write(data)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
146
35289
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
147 # XXX: should we verify the content of the cache, and hardlink back to
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
148 # the local store on success, but truncate, write and link on failure?
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
149 if not self.cachevfs.exists(oid):
35553
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
150 self.ui.note(_('lfs: adding %s to the usercache\n') % oid)
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
151 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
35289
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
152
35480
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
153 def read(self, oid, verify=True):
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
154 """Read blob from local blobstore."""
35289
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
155 if not self.vfs.exists(oid):
35480
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
156 blob = self._read(self.cachevfs, oid, verify)
35481
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35480
diff changeset
157
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35480
diff changeset
158 # Even if revlog will verify the content, it needs to be verified
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35480
diff changeset
159 # now before making the hardlink to avoid propagating corrupt blobs.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35480
diff changeset
160 # Don't abort if corruption is detected, because `hg verify` will
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35480
diff changeset
161 # give more useful info about the corruption- simply don't add the
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35480
diff changeset
162 # hardlink.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35480
diff changeset
163 if verify or hashlib.sha256(blob).hexdigest() == oid:
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35480
diff changeset
164 self.ui.note(_('lfs: found %s in the usercache\n') % oid)
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35480
diff changeset
165 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
35477
02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35444
diff changeset
166 else:
02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35444
diff changeset
167 self.ui.note(_('lfs: found %s in the local lfs store\n') % oid)
35480
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
168 blob = self._read(self.vfs, oid, verify)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
169 return blob
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
170
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
171 def _read(self, vfs, oid, verify):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
172 """Read blob (after verifying) from the given store"""
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
173 blob = vfs.read(oid)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
174 if verify:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
175 _verify(oid, blob)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
176 return blob
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
177
37148
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
178 def verify(self, oid):
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
179 """Indicate whether or not the hash of the underlying file matches its
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
180 name."""
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
181 sha256 = hashlib.sha256()
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
182
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
183 with self.open(oid) as fp:
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
184 for chunk in util.filechunkiter(fp, size=1048576):
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
185 sha256.update(chunk)
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
186
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
187 return oid == sha256.hexdigest()
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36932
diff changeset
188
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
189 def has(self, oid):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
190 """Returns True if the local blobstore contains the requested blob,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
191 False otherwise."""
35289
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35123
diff changeset
192 return self.cachevfs.exists(oid) or self.vfs.exists(oid)
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
193
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
194 class _gitlfsremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
195
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
196 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
197 ui = repo.ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
198 self.ui = ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
199 baseurl, authinfo = url.authinfo()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
200 self.baseurl = baseurl.rstrip('/')
35444
e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35443
diff changeset
201 useragent = repo.ui.config('experimental', 'lfs.user-agent')
e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35443
diff changeset
202 if not useragent:
35733
3d48ae1aaa5e lfs: default the User-Agent header for blob transfers to 'git-lfs'
Matt Harbison <matt_harbison@yahoo.com>
parents: 35732
diff changeset
203 useragent = 'git-lfs/2.3.4 (Mercurial %s)' % util.version()
35443
e7bb5fc4570c lfs: add git to the User-Agent header for blob transfers
Matt Harbison <matt_harbison@yahoo.com>
parents: 35437
diff changeset
204 self.urlopener = urlmod.opener(ui, authinfo, useragent)
35123
07e97998d385 lfs: register config options
Matt Harbison <matt_harbison@yahoo.com>
parents: 35122
diff changeset
205 self.retry = ui.configint('lfs', 'retry')
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
206
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
207 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
208 """Batch upload from local to remote blobstore."""
35967
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
209 self._batch(_deduplicate(pointers), fromstore, 'upload')
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
210
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
211 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
212 """Batch download from remote to local blostore."""
35967
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
213 self._batch(_deduplicate(pointers), tostore, 'download')
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
214
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
215 def _batchrequest(self, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
216 """Get metadata about objects pointed by pointers for given action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
217
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
218 Return decoded JSON object like {'objects': [{'oid': '', 'size': 1}]}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
219 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
220 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
221 objects = [{'oid': p.oid(), 'size': p.size()} for p in pointers]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
222 requestdata = json.dumps({
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
223 'objects': objects,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
224 'operation': action,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
225 })
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
226 batchreq = util.urlreq.request('%s/objects/batch' % self.baseurl,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
227 data=requestdata)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
228 batchreq.add_header('Accept', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
229 batchreq.add_header('Content-Type', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
230 try:
36932
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
231 rsp = self.urlopener.open(batchreq)
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
232 rawjson = rsp.read()
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
233 except util.urlerr.httperror as ex:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
234 raise LfsRemoteError(_('LFS HTTP error: %s (action=%s)')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
235 % (ex, action))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
236 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
237 response = json.loads(rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
238 except ValueError:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
239 raise LfsRemoteError(_('LFS server returns invalid JSON: %s')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
240 % rawjson)
36932
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
241
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
242 if self.ui.debugflag:
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
243 self.ui.debug('Status: %d\n' % rsp.status)
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
244 # lfs-test-server and hg serve return headers in different order
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
245 self.ui.debug('%s\n'
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
246 % '\n'.join(sorted(str(rsp.info()).splitlines())))
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
247
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
248 if 'objects' in response:
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
249 response['objects'] = sorted(response['objects'],
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
250 key=lambda p: p['oid'])
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
251 self.ui.debug('%s\n'
37149
c37c47e47a95 test-lfs: drop trailing ', ' item separators from debug JSON output
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
252 % json.dumps(response, indent=2,
c37c47e47a95 test-lfs: drop trailing ', ' item separators from debug JSON output
Matt Harbison <matt_harbison@yahoo.com>
parents: 37148
diff changeset
253 separators=('', ': '), sort_keys=True))
36932
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
254
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
255 return response
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
256
35666
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
257 def _checkforservererror(self, pointers, responses, action):
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
258 """Scans errors from objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
259
35694
8a23082f4d93 lfs: correct documentation typo
Matt Harbison <matt_harbison@yahoo.com>
parents: 35666
diff changeset
260 Raises LfsRemoteError if any objects have an error"""
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
261 for response in responses:
35666
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
262 # The server should return 404 when objects cannot be found. Some
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
263 # server implementation (ex. lfs-test-server) does not set "error"
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
264 # but just removes "download" from "actions". Treat that case
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
265 # as the same as 404 error.
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
266 notfound = (response.get('error', {}).get('code') == 404
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
267 or (action == 'download'
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
268 and action not in response.get('actions', [])))
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
269 if notfound:
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
270 ptrmap = {p.oid(): p for p in pointers}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
271 p = ptrmap.get(response['oid'], None)
35666
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
272 if p:
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
273 filename = getattr(p, 'filename', 'unknown')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
274 raise LfsRemoteError(
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
275 _(('LFS server error. Remote object '
35568
ebf14075a5c1 lfs: improve the error message for a missing remote blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35553
diff changeset
276 'for "%s" not found: %r')) % (filename, response))
35695
dd672e3d059f lfs: raise an error if the server sends an unsolicited oid
Matt Harbison <matt_harbison@yahoo.com>
parents: 35694
diff changeset
277 else:
dd672e3d059f lfs: raise an error if the server sends an unsolicited oid
Matt Harbison <matt_harbison@yahoo.com>
parents: 35694
diff changeset
278 raise LfsRemoteError(
dd672e3d059f lfs: raise an error if the server sends an unsolicited oid
Matt Harbison <matt_harbison@yahoo.com>
parents: 35694
diff changeset
279 _('LFS server error. Unsolicited response for oid %s')
dd672e3d059f lfs: raise an error if the server sends an unsolicited oid
Matt Harbison <matt_harbison@yahoo.com>
parents: 35694
diff changeset
280 % response['oid'])
35666
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
281 if 'error' in response:
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
282 raise LfsRemoteError(_('LFS server error: %r') % response)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
283
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
284 def _extractobjects(self, response, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
285 """extract objects from response of the batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
286
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
287 response: parsed JSON object returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
288 return response['objects'] filtered by action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
289 raise if any object has an error
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
290 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
291 # Scan errors from objects - fail early
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
292 objects = response.get('objects', [])
35666
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
293 self._checkforservererror(pointers, objects, action)
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
294
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
295 # Filter objects with given action. Practically, this skips uploading
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
296 # objects which exist in the server.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
297 filteredobjects = [o for o in objects if action in o.get('actions', [])]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
298
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
299 return filteredobjects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
300
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
301 def _basictransfer(self, obj, action, localstore):
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
302 """Download or upload a single object using basic transfer protocol
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
303
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
304 obj: dict, an object description returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
305 action: string, one of ['upload', 'download']
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
306 localstore: blobstore.local
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
307
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
308 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/\
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
309 basic-transfers.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
310 """
36630
4da09b46451e lfs: add some bytestring wrappers in blobstore.py
Augie Fackler <augie@google.com>
parents: 36485
diff changeset
311 oid = pycompat.bytestr(obj['oid'])
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
312
36630
4da09b46451e lfs: add some bytestring wrappers in blobstore.py
Augie Fackler <augie@google.com>
parents: 36485
diff changeset
313 href = pycompat.bytestr(obj['actions'][action].get('href'))
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
314 headers = obj['actions'][action].get('header', {}).items()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
315
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
316 request = util.urlreq.request(href)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
317 if action == 'upload':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
318 # If uploading blobs, read data from local blobstore.
35530
e8f80529abeb lfs: use the local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35529
diff changeset
319 with localstore.open(oid) as fp:
35480
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
320 _verifyfile(oid, fp)
35530
e8f80529abeb lfs: use the local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35529
diff changeset
321 request.data = filewithprogress(localstore.open(oid), None)
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
322 request.get_method = lambda: 'PUT'
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
323
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
324 for k, v in headers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
325 request.add_header(k, v)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
326
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
327 response = b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
328 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
329 req = self.urlopener.open(request)
36932
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
330
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
331 if self.ui.debugflag:
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
332 self.ui.debug('Status: %d\n' % req.status)
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
333 # lfs-test-server and hg serve return headers in different order
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
334 self.ui.debug('%s\n'
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
335 % '\n'.join(sorted(str(req.info()).splitlines())))
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36630
diff changeset
336
35552
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
337 if action == 'download':
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
338 # If downloading blobs, store downloaded data to local blobstore
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
339 localstore.download(oid, req)
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
340 else:
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
341 while True:
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
342 data = req.read(1048576)
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
343 if not data:
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
344 break
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
345 response += data
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
346 if response:
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
347 self.ui.debug('lfs %s response: %s' % (action, response))
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
348 except util.urlerr.httperror as ex:
35734
b4e1d0654736 lfs: dump the full response on httperror in debug mode
Matt Harbison <matt_harbison@yahoo.com>
parents: 35733
diff changeset
349 if self.ui.debugflag:
35753
069df0b952e8 lfs: separate a debug message from the subsequent abort message
Matt Harbison <matt_harbison@yahoo.com>
parents: 35734
diff changeset
350 self.ui.debug('%s: %s\n' % (oid, ex.read()))
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
351 raise LfsRemoteError(_('HTTP error: %s (oid=%s, action=%s)')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
352 % (ex, oid, action))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
353
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
354 def _batch(self, pointers, localstore, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
355 if action not in ['upload', 'download']:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
356 raise error.ProgrammingError('invalid Git-LFS action: %s' % action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
357
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
358 response = self._batchrequest(pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
359 objects = self._extractobjects(response, pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
360 total = sum(x.get('size', 0) for x in objects)
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
361 sizes = {}
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
362 for obj in objects:
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
363 sizes[obj.get('oid')] = obj.get('size', 0)
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
364 topic = {'upload': _('lfs uploading'),
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
365 'download': _('lfs downloading')}[action]
35482
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35481
diff changeset
366 if len(objects) > 1:
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35481
diff changeset
367 self.ui.note(_('lfs: need to transfer %d objects (%s)\n')
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35481
diff changeset
368 % (len(objects), util.bytecount(total)))
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
369 self.ui.progress(topic, 0, total=total)
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
370 def transfer(chunk):
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
371 for obj in chunk:
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
372 objsize = obj.get('size', 0)
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
373 if self.ui.verbose:
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
374 if action == 'download':
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
375 msg = _('lfs: downloading %s (%s)\n')
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
376 elif action == 'upload':
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
377 msg = _('lfs: uploading %s (%s)\n')
35482
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35481
diff changeset
378 self.ui.note(msg % (obj.get('oid'),
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35481
diff changeset
379 util.bytecount(objsize)))
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
380 retry = self.retry
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
381 while True:
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
382 try:
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
383 self._basictransfer(obj, action, localstore)
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
384 yield 1, obj.get('oid')
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
385 break
35479
b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry
Matt Harbison <matt_harbison@yahoo.com>
parents: 35477
diff changeset
386 except socket.error as ex:
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
387 if retry > 0:
35482
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35481
diff changeset
388 self.ui.note(
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35481
diff changeset
389 _('lfs: failed: %r (remaining retry %d)\n')
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35481
diff changeset
390 % (ex, retry))
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
391 retry -= 1
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
392 continue
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
393 raise
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
394
35732
10e62d5efa73 lfs: default to not using workers for upload/download
Matt Harbison <matt_harbison@yahoo.com>
parents: 35695
diff changeset
395 # Until https multiplexing gets sorted out
10e62d5efa73 lfs: default to not using workers for upload/download
Matt Harbison <matt_harbison@yahoo.com>
parents: 35695
diff changeset
396 if self.ui.configbool('experimental', 'lfs.worker-enable'):
10e62d5efa73 lfs: default to not using workers for upload/download
Matt Harbison <matt_harbison@yahoo.com>
parents: 35695
diff changeset
397 oids = worker.worker(self.ui, 0.1, transfer, (),
10e62d5efa73 lfs: default to not using workers for upload/download
Matt Harbison <matt_harbison@yahoo.com>
parents: 35695
diff changeset
398 sorted(objects, key=lambda o: o.get('oid')))
10e62d5efa73 lfs: default to not using workers for upload/download
Matt Harbison <matt_harbison@yahoo.com>
parents: 35695
diff changeset
399 else:
10e62d5efa73 lfs: default to not using workers for upload/download
Matt Harbison <matt_harbison@yahoo.com>
parents: 35695
diff changeset
400 oids = transfer(sorted(objects, key=lambda o: o.get('oid')))
10e62d5efa73 lfs: default to not using workers for upload/download
Matt Harbison <matt_harbison@yahoo.com>
parents: 35695
diff changeset
401
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
402 processed = 0
35921
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
403 blobs = 0
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
404 for _one, oid in oids:
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
405 processed += sizes[oid]
35921
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
406 blobs += 1
35437
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35403
diff changeset
407 self.ui.progress(topic, processed, total=total)
35482
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35481
diff changeset
408 self.ui.note(_('lfs: processed: %s\n') % oid)
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
409 self.ui.progress(topic, pos=None, total=total)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
410
35921
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
411 if blobs > 0:
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
412 if action == 'upload':
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
413 self.ui.status(_('lfs: uploaded %d files (%s)\n')
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
414 % (blobs, util.bytecount(processed)))
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
415 # TODO: coalesce the download requests, and comment this in
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
416 #elif action == 'download':
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
417 # self.ui.status(_('lfs: downloaded %d files (%s)\n')
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
418 # % (blobs, util.bytecount(processed)))
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
419
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
420 def __del__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
421 # copied from mercurial/httppeer.py
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
422 urlopener = getattr(self, 'urlopener', None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
423 if urlopener:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
424 for h in urlopener.handlers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
425 h.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
426 getattr(h, "close_all", lambda : None)()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
427
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
428 class _dummyremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
429 """Dummy store storing blobs to temp directory."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
430
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
431 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
432 fullpath = repo.vfs.join('lfs', url.path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
433 self.vfs = lfsvfs(fullpath)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
434
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
435 def writebatch(self, pointers, fromstore):
35967
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
436 for p in _deduplicate(pointers):
35480
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
437 content = fromstore.read(p.oid(), verify=True)
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
438 with self.vfs(p.oid(), 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
439 fp.write(content)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
440
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
441 def readbatch(self, pointers, tostore):
35967
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
442 for p in _deduplicate(pointers):
35552
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
443 with self.vfs(p.oid(), 'rb') as fp:
fd610befc37f lfs: use the localstore download method to transfer from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35551
diff changeset
444 tostore.download(p.oid(), fp)
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
445
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
446 class _nullremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
447 """Null store storing blobs to /dev/null."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
448
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
449 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
450 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
451
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
452 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
453 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
454
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
455 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
456 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
457
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
458 class _promptremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
459 """Prompt user to set lfs.url when accessed."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
460
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
461 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
462 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
463
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
464 def writebatch(self, pointers, fromstore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
465 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
466
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
467 def readbatch(self, pointers, tostore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
468 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
469
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
470 def _prompt(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
471 raise error.Abort(_('lfs.url needs to be configured'))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
472
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
473 _storemap = {
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
474 'https': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
475 'http': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
476 'file': _dummyremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
477 'null': _nullremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
478 None: _promptremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
479 }
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
480
35967
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
481 def _deduplicate(pointers):
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
482 """Remove any duplicate oids that exist in the list"""
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
483 reduced = util.sortdict()
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
484 for p in pointers:
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
485 reduced[p.oid()] = p
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
486 return reduced.values()
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35921
diff changeset
487
35480
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
488 def _verify(oid, content):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
489 realoid = hashlib.sha256(content).hexdigest()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
490 if realoid != oid:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
491 raise error.Abort(_('detected corrupt lfs object: %s') % oid,
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
492 hint=_('run hg verify'))
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
493
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
494 def _verifyfile(oid, fp):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
495 sha256 = hashlib.sha256()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
496 while True:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
497 data = fp.read(1024 * 1024)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
498 if not data:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
499 break
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
500 sha256.update(data)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
501 realoid = sha256.hexdigest()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
502 if realoid != oid:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
503 raise error.Abort(_('detected corrupt lfs object: %s') % oid,
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
504 hint=_('run hg verify'))
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35479
diff changeset
505
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
506 def remote(repo):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
507 """remotestore factory. return a store in _storemap depending on config"""
35614
6d6d20658cce lfs: drop deprecated remote store config options
Matt Harbison <matt_harbison@yahoo.com>
parents: 35568
diff changeset
508 url = util.url(repo.ui.config('lfs', 'url') or '')
35121
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
509 scheme = url.scheme
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
510 if scheme not in _storemap:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
511 raise error.Abort(_('lfs: unknown url scheme: %s') % scheme)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
512 return _storemap[scheme](repo, url)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
513
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
514 class LfsRemoteError(error.RevlogError):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
515 pass