mercurial-scm/hg-stable: hgext/lfs/blobstore.py annotate

annotate hgext/lfs/blobstore.py @ 40703:93e5d18251d6

lfs: make the exception messages consistent I don't love that it repeats 'HTTP Error' in an already long message, but I doubt that we should assume that it will always say that on the original exception message.

author	Matt Harbison <matt_harbison@yahoo.com>
date	Thu, 15 Nov 2018 18:14:57 -0500
parents	380f5131ee7b
children	fb379b78b93e

rev	line source
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	1 # blobstore.py - local and remote (speaking Git-LFS protocol) blob storages
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	2 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	3 # Copyright 2017 Facebook, Inc.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	4 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	7
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	8 from __future__ import absolute_import
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	9
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	10 import errno
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	11 import hashlib
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	12 import json
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	13 import os
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	14 import re
35479 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	15 import socket
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	16
35122 b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35121 diff changeset	17 from mercurial.i18n import _
b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35121 diff changeset	18
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	19 from mercurial import (
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	20 encoding,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	21 error,
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	22 pathutil,
36630 4da09b46451e lfs: add some bytestring wrappers in blobstore.py Augie Fackler <augie@google.com> parents: 36485 diff changeset	23 pycompat,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	24 url as urlmod,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	25 util,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	26 vfs as vfsmod,
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	27 worker,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	28 )
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	29
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	30 from mercurial.utils import (
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	31 stringutil,
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	32 )
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	33
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	34 from ..largefiles import lfutil
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	35
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	36 # 64 bytes for SHA256
36485 9e3cb58c7ab3 py3: make sure regexes are bytes Pulkit Goyal <7895pulkit@gmail.com> parents: 35967 diff changeset	37 _lfsre = re.compile(br'\A[a-f0-9]{64}\Z')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	38
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	39 class lfsvfs(vfsmod.vfs):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	40 def join(self, path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	41 """split the path at first two characters, like: XX/XXXXX..."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	42 if not _lfsre.match(path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	43 raise error.ProgrammingError('unexpected lfs path: %s' % path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	44 return super(lfsvfs, self).join(path[0:2], path[2:])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	45
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	46 def walk(self, path=None, onerror=None):
35403 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35371 diff changeset	47 """Yield (dirpath, [], oids) tuple for blobs under path
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	48
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	49 Oids only exist in the root of this vfs, so dirpath is always ''.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	50 """
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	51 root = os.path.normpath(self.base)
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	52 # when dirpath == root, dirpath[prefixlen:] becomes empty
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	53 # because len(dirpath) < prefixlen.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	54 prefixlen = len(pathutil.normasprefix(root))
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	55 oids = []
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	56
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	57 for dirpath, dirs, files in os.walk(self.reljoin(self.base, path or ''),
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	58 onerror=onerror):
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	59 dirpath = dirpath[prefixlen:]
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	60
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	61 # Silently skip unexpected files and directories
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	62 if len(dirpath) == 2:
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	63 oids.extend([dirpath + f for f in files
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	64 if _lfsre.match(dirpath + f)])
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	65
35403 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35371 diff changeset	66 yield ('', [], oids)
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	67
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	68 class nullvfs(lfsvfs):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	69 def __init__(self):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	70 pass
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	71
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	72 def exists(self, oid):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	73 return False
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	74
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	75 def read(self, oid):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	76 # store.read() calls into here if the blob doesn't exist in its
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	77 # self.vfs. Raise the same error as a normal vfs when asked to read a
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	78 # file that doesn't exist. The only difference is the full file path
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	79 # isn't available in the error.
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	80 raise IOError(errno.ENOENT, '%s: No such file or directory' % oid)
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	81
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	82 def walk(self, path=None, onerror=None):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	83 return ('', [], [])
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	84
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	85 def write(self, oid, data):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	86 pass
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	87
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	88 class filewithprogress(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	89 """a file-like object that supports __len__ and read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	90
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	91 Useful to provide progress information for how many bytes are read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	92 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	93
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	94 def __init__(self, fp, callback):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	95 self._fp = fp
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	96 self._callback = callback # func(readsize)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	97 fp.seek(0, os.SEEK_END)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	98 self._len = fp.tell()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	99 fp.seek(0)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	100
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	101 def __len__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	102 return self._len
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	103
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	104 def read(self, size):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	105 if self._fp is None:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	106 return b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	107 data = self._fp.read(size)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	108 if data:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	109 if self._callback:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	110 self._callback(len(data))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	111 else:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	112 self._fp.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	113 self._fp = None
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	114 return data
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	115
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	116 class local(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	117 """Local blobstore for large file contents.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	118
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	119 This blobstore is used both as a cache and as a staging area for large blobs
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	120 to be uploaded to the remote blobstore.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	121 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	122
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	123 def __init__(self, repo):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	124 fullpath = repo.svfs.join('lfs/objects')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	125 self.vfs = lfsvfs(fullpath)
37562 e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url Matt Harbison <matt_harbison@yahoo.com> parents: 37518 diff changeset	126
e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url Matt Harbison <matt_harbison@yahoo.com> parents: 37518 diff changeset	127 if repo.ui.configbool('experimental', 'lfs.disableusercache'):
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	128 self.cachevfs = nullvfs()
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	129 else:
37562 e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url Matt Harbison <matt_harbison@yahoo.com> parents: 37518 diff changeset	130 usercache = lfutil._usercachedir(repo.ui, 'lfs')
e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url Matt Harbison <matt_harbison@yahoo.com> parents: 37518 diff changeset	131 self.cachevfs = lfsvfs(usercache)
35477 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35444 diff changeset	132 self.ui = repo.ui
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	133
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	134 def open(self, oid):
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	135 """Open a read-only file descriptor to the named blob, in either the
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	136 usercache or the local store."""
35541 58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	137 # The usercache is the most likely place to hold the file. Commit will
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	138 # write to both it and the local store, as will anything that downloads
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	139 # the blobs. However, things like clone without an update won't
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	140 # populate the local store. For an init + push of a local clone,
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	141 # the usercache is the only place it _could_ be. If not present, the
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	142 # missing file msg here will indicate the local repo, not the usercache.
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	143 if self.cachevfs.exists(oid):
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	144 return self.cachevfs(oid, 'rb')
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	145
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	146 return self.vfs(oid, 'rb')
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	147
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	148 def download(self, oid, src):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	149 """Read the blob from the remote source in chunks, verify the content,
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	150 and write to this local blobstore."""
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	151 sha256 = hashlib.sha256()
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	152
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	153 with self.vfs(oid, 'wb', atomictemp=True) as fp:
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	154 for chunk in util.filechunkiter(src, size=1048576):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	155 fp.write(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	156 sha256.update(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	157
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	158 realoid = sha256.hexdigest()
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	159 if realoid != oid:
37692 10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	160 raise LfsCorruptionError(_('corrupt remote lfs object: %s')
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	161 % oid)
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	162
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	163 self._linktousercache(oid)
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	164
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	165 def write(self, oid, data):
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	166 """Write blob to local blobstore.
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	167
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	168 This should only be called from the filelog during a commit or similar.
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	169 As such, there is no need to verify the data. Imports from a remote
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	170 store must use ``download()`` instead."""
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	171 with self.vfs(oid, 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	172 fp.write(data)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	173
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	174 self._linktousercache(oid)
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	175
39474 a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	176 def linkfromusercache(self, oid):
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	177 """Link blobs found in the user cache into this store.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	178
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	179 The server module needs to do this when it lets the client know not to
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	180 upload the blob, to ensure it is always available in this store.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	181 Normally this is done implicitly when the client reads or writes the
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	182 blob, but that doesn't happen when the server tells the client that it
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	183 already has the blob.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	184 """
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	185 if (not isinstance(self.cachevfs, nullvfs)
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	186 and not self.vfs.exists(oid)):
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	187 self.ui.note(_('lfs: found %s in the usercache\n') % oid)
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	188 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	189
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	190 def _linktousercache(self, oid):
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	191 # XXX: should we verify the content of the cache, and hardlink back to
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	192 # the local store on success, but truncate, write and link on failure?
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	193 if (not self.cachevfs.exists(oid)
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	194 and not isinstance(self.cachevfs, nullvfs)):
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	195 self.ui.note(_('lfs: adding %s to the usercache\n') % oid)
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	196 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	197
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	198 def read(self, oid, verify=True):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	199 """Read blob from local blobstore."""
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	200 if not self.vfs.exists(oid):
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	201 blob = self._read(self.cachevfs, oid, verify)
35481 bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	202
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	203 # Even if revlog will verify the content, it needs to be verified
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	204 # now before making the hardlink to avoid propagating corrupt blobs.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	205 # Don't abort if corruption is detected, because `hg verify` will
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	206 # give more useful info about the corruption- simply don't add the
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	207 # hardlink.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	208 if verify or hashlib.sha256(blob).hexdigest() == oid:
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	209 self.ui.note(_('lfs: found %s in the usercache\n') % oid)
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	210 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
35477 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35444 diff changeset	211 else:
02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35444 diff changeset	212 self.ui.note(_('lfs: found %s in the local lfs store\n') % oid)
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	213 blob = self._read(self.vfs, oid, verify)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	214 return blob
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	215
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	216 def _read(self, vfs, oid, verify):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	217 """Read blob (after verifying) from the given store"""
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	218 blob = vfs.read(oid)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	219 if verify:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	220 _verify(oid, blob)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	221 return blob
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	222
37148 56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	223 def verify(self, oid):
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	224 """Indicate whether or not the hash of the underlying file matches its
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	225 name."""
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	226 sha256 = hashlib.sha256()
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	227
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	228 with self.open(oid) as fp:
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	229 for chunk in util.filechunkiter(fp, size=1048576):
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	230 sha256.update(chunk)
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	231
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	232 return oid == sha256.hexdigest()
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	233
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	234 def has(self, oid):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	235 """Returns True if the local blobstore contains the requested blob,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	236 False otherwise."""
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	237 return self.cachevfs.exists(oid) or self.vfs.exists(oid)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	238
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	239 def _urlerrorreason(urlerror):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	240 '''Create a friendly message for the given URLError to be used in an
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	241 LfsRemoteError message.
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	242 '''
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	243 inst = urlerror
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	244
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	245 if isinstance(urlerror.reason, Exception):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	246 inst = urlerror.reason
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	247
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	248 if util.safehasattr(inst, 'reason'):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	249 try: # usually it is in the form (errno, strerror)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	250 reason = inst.reason.args[1]
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	251 except (AttributeError, IndexError):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	252 # it might be anything, for example a string
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	253 reason = inst.reason
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	254 if isinstance(reason, pycompat.unicode):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	255 # SSLError of Python 2.7.9 contains a unicode
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	256 reason = encoding.unitolocal(reason)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	257 return reason
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	258 elif getattr(inst, "strerror", None):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	259 return encoding.strtolocal(inst.strerror)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	260 else:
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	261 return stringutil.forcebytestr(urlerror)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	262
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	263 class _gitlfsremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	264
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	265 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	266 ui = repo.ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	267 self.ui = ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	268 baseurl, authinfo = url.authinfo()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	269 self.baseurl = baseurl.rstrip('/')
35444 e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35443 diff changeset	270 useragent = repo.ui.config('experimental', 'lfs.user-agent')
e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35443 diff changeset	271 if not useragent:
35733 3d48ae1aaa5e lfs: default the User-Agent header for blob transfers to 'git-lfs' Matt Harbison <matt_harbison@yahoo.com> parents: 35732 diff changeset	272 useragent = 'git-lfs/2.3.4 (Mercurial %s)' % util.version()
35443 e7bb5fc4570c lfs: add git to the User-Agent header for blob transfers Matt Harbison <matt_harbison@yahoo.com> parents: 35437 diff changeset	273 self.urlopener = urlmod.opener(ui, authinfo, useragent)
35123 07e97998d385 lfs: register config options Matt Harbison <matt_harbison@yahoo.com> parents: 35122 diff changeset	274 self.retry = ui.configint('lfs', 'retry')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	275
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	276 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	277 """Batch upload from local to remote blobstore."""
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	278 self._batch(_deduplicate(pointers), fromstore, 'upload')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	279
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	280 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	281 """Batch download from remote to local blostore."""
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	282 self._batch(_deduplicate(pointers), tostore, 'download')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	283
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	284 def _batchrequest(self, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	285 """Get metadata about objects pointed by pointers for given action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	286
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	287 Return decoded JSON object like {'objects': [{'oid': '', 'size': 1}]}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	288 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	289 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	290 objects = [{'oid': p.oid(), 'size': p.size()} for p in pointers]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	291 requestdata = json.dumps({
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	292 'objects': objects,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	293 'operation': action,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	294 })
40700 8863f08c1630 lfs: provide more Batch API error info via a hint in the raised exception Matt Harbison <matt_harbison@yahoo.com> parents: 39793 diff changeset	295 url = '%s/objects/batch' % self.baseurl
8863f08c1630 lfs: provide more Batch API error info via a hint in the raised exception Matt Harbison <matt_harbison@yahoo.com> parents: 39793 diff changeset	296 batchreq = util.urlreq.request(url, data=requestdata)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	297 batchreq.add_header('Accept', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	298 batchreq.add_header('Content-Type', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	299 try:
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	300 rsp = self.urlopener.open(batchreq)
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	301 rawjson = rsp.read()
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	302 except util.urlerr.httperror as ex:
40701 9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	303 hints = {
9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	304 400: _('check that lfs serving is enabled on %s and "%s" is '
9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	305 'supported') % (self.baseurl, action),
9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	306 404: _('the "lfs.url" config may be used to override %s')
9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	307 % self.baseurl,
9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	308 }
9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	309 hint = hints.get(ex.code, _('api=%s, action=%s') % (url, action))
9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	310 raise LfsRemoteError(_('LFS HTTP error: %s') % ex, hint=hint)
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	311 except util.urlerr.urlerror as ex:
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	312 hint = (_('the "lfs.url" config may be used to override %s')
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	313 % self.baseurl)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	314 raise LfsRemoteError(_('LFS error: %s') % _urlerrorreason(ex),
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	315 hint=hint)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	316 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	317 response = json.loads(rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	318 except ValueError:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	319 raise LfsRemoteError(_('LFS server returns invalid JSON: %s')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	320 % rawjson)
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	321
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	322 if self.ui.debugflag:
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	323 self.ui.debug('Status: %d\n' % rsp.status)
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	324 # lfs-test-server and hg serve return headers in different order
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	325 self.ui.debug('%s\n'
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	326 % '\n'.join(sorted(str(rsp.info()).splitlines())))
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	327
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	328 if 'objects' in response:
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	329 response['objects'] = sorted(response['objects'],
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	330 key=lambda p: p['oid'])
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	331 self.ui.debug('%s\n'
37149 c37c47e47a95 test-lfs: drop trailing ', ' item separators from debug JSON output Matt Harbison <matt_harbison@yahoo.com> parents: 37148 diff changeset	332 % json.dumps(response, indent=2,
c37c47e47a95 test-lfs: drop trailing ', ' item separators from debug JSON output Matt Harbison <matt_harbison@yahoo.com> parents: 37148 diff changeset	333 separators=('', ': '), sort_keys=True))
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	334
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	335 return response
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	336
35666 2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	337 def _checkforservererror(self, pointers, responses, action):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	338 """Scans errors from objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	339
35694 8a23082f4d93 lfs: correct documentation typo Matt Harbison <matt_harbison@yahoo.com> parents: 35666 diff changeset	340 Raises LfsRemoteError if any objects have an error"""
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	341 for response in responses:
35666 2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	342 # The server should return 404 when objects cannot be found. Some
2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	343 # server implementation (ex. lfs-test-server) does not set "error"
2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	344 # but just removes "download" from "actions". Treat that case
2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	345 # as the same as 404 error.
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	346 if 'error' not in response:
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	347 if (action == 'download'
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	348 and action not in response.get('actions', [])):
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	349 code = 404
35695 dd672e3d059f lfs: raise an error if the server sends an unsolicited oid Matt Harbison <matt_harbison@yahoo.com> parents: 35694 diff changeset	350 else:
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	351 continue
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	352 else:
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	353 # An error dict without a code doesn't make much sense, so
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	354 # treat as a server error.
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	355 code = response.get('error').get('code', 500)
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	356
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	357 ptrmap = {p.oid(): p for p in pointers}
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	358 p = ptrmap.get(response['oid'], None)
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	359 if p:
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	360 filename = getattr(p, 'filename', 'unknown')
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	361 errors = {
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	362 404: 'The object does not exist',
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	363 410: 'The object was removed by the owner',
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	364 422: 'Validation error',
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	365 500: 'Internal server error',
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	366 }
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	367 msg = errors.get(code, 'status code %d' % code)
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	368 raise LfsRemoteError(_('LFS server error for "%s": %s')
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	369 % (filename, msg))
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	370 else:
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	371 raise LfsRemoteError(
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	372 _('LFS server error. Unsolicited response for oid %s')
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	373 % response['oid'])
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	374
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	375 def _extractobjects(self, response, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	376 """extract objects from response of the batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	377
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	378 response: parsed JSON object returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	379 return response['objects'] filtered by action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	380 raise if any object has an error
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	381 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	382 # Scan errors from objects - fail early
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	383 objects = response.get('objects', [])
35666 2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	384 self._checkforservererror(pointers, objects, action)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	385
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	386 # Filter objects with given action. Practically, this skips uploading
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	387 # objects which exist in the server.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	388 filteredobjects = [o for o in objects if action in o.get('actions', [])]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	389
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	390 return filteredobjects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	391
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	392 def _basictransfer(self, obj, action, localstore):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	393 """Download or upload a single object using basic transfer protocol
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	394
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	395 obj: dict, an object description returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	396 action: string, one of ['upload', 'download']
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	397 localstore: blobstore.local
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	398
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	399 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/\
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	400 basic-transfers.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	401 """
36630 4da09b46451e lfs: add some bytestring wrappers in blobstore.py Augie Fackler <augie@google.com> parents: 36485 diff changeset	402 oid = pycompat.bytestr(obj['oid'])
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	403
36630 4da09b46451e lfs: add some bytestring wrappers in blobstore.py Augie Fackler <augie@google.com> parents: 36485 diff changeset	404 href = pycompat.bytestr(obj['actions'][action].get('href'))
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	405 headers = obj['actions'][action].get('header', {}).items()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	406
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	407 request = util.urlreq.request(href)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	408 if action == 'upload':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	409 # If uploading blobs, read data from local blobstore.
37219 b00bd974eef5 lfs: drop a duplicate blob verification method Matt Harbison <matt_harbison@yahoo.com> parents: 37149 diff changeset	410 if not localstore.verify(oid):
b00bd974eef5 lfs: drop a duplicate blob verification method Matt Harbison <matt_harbison@yahoo.com> parents: 37149 diff changeset	411 raise error.Abort(_('detected corrupt lfs object: %s') % oid,
b00bd974eef5 lfs: drop a duplicate blob verification method Matt Harbison <matt_harbison@yahoo.com> parents: 37149 diff changeset	412 hint=_('run hg verify'))
35530 e8f80529abeb lfs: use the local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35529 diff changeset	413 request.data = filewithprogress(localstore.open(oid), None)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	414 request.get_method = lambda: 'PUT'
37245 3e293808e835 lfs: add the 'Content-Type' header called out in the file transfer spec Matt Harbison <matt_harbison@yahoo.com> parents: 37244 diff changeset	415 request.add_header('Content-Type', 'application/octet-stream')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	416
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	417 for k, v in headers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	418 request.add_header(k, v)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	419
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	420 response = b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	421 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	422 req = self.urlopener.open(request)
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	423
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	424 if self.ui.debugflag:
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	425 self.ui.debug('Status: %d\n' % req.status)
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	426 # lfs-test-server and hg serve return headers in different order
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	427 self.ui.debug('%s\n'
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	428 % '\n'.join(sorted(str(req.info()).splitlines())))
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	429
35552 fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	430 if action == 'download':
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	431 # If downloading blobs, store downloaded data to local blobstore
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	432 localstore.download(oid, req)
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	433 else:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	434 while True:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	435 data = req.read(1048576)
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	436 if not data:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	437 break
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	438 response += data
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	439 if response:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	440 self.ui.debug('lfs %s response: %s' % (action, response))
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	441 except util.urlerr.httperror as ex:
35734 b4e1d0654736 lfs: dump the full response on httperror in debug mode Matt Harbison <matt_harbison@yahoo.com> parents: 35733 diff changeset	442 if self.ui.debugflag:
35753 069df0b952e8 lfs: separate a debug message from the subsequent abort message Matt Harbison <matt_harbison@yahoo.com> parents: 35734 diff changeset	443 self.ui.debug('%s: %s\n' % (oid, ex.read()))
40703 93e5d18251d6 lfs: make the exception messages consistent Matt Harbison <matt_harbison@yahoo.com> parents: 40702 diff changeset	444 raise LfsRemoteError(_('LFS HTTP error: %s (oid=%s, action=%s)')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	445 % (ex, oid, action))
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	446 except util.urlerr.urlerror as ex:
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	447 hint = (_('attempted connection to %s')
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	448 % util.urllibcompat.getfullurl(request))
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	449 raise LfsRemoteError(_('LFS error: %s') % _urlerrorreason(ex),
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	450 hint=hint)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	451
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	452 def _batch(self, pointers, localstore, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	453 if action not in ['upload', 'download']:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	454 raise error.ProgrammingError('invalid Git-LFS action: %s' % action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	455
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	456 response = self._batchrequest(pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	457 objects = self._extractobjects(response, pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	458 total = sum(x.get('size', 0) for x in objects)
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	459 sizes = {}
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	460 for obj in objects:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	461 sizes[obj.get('oid')] = obj.get('size', 0)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	462 topic = {'upload': _('lfs uploading'),
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	463 'download': _('lfs downloading')}[action]
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	464 if len(objects) > 1:
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	465 self.ui.note(_('lfs: need to transfer %d objects (%s)\n')
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	466 % (len(objects), util.bytecount(total)))
39416 b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	467
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	468 def transfer(chunk):
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	469 for obj in chunk:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	470 objsize = obj.get('size', 0)
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	471 if self.ui.verbose:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	472 if action == 'download':
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	473 msg = _('lfs: downloading %s (%s)\n')
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	474 elif action == 'upload':
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	475 msg = _('lfs: uploading %s (%s)\n')
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	476 self.ui.note(msg % (obj.get('oid'),
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	477 util.bytecount(objsize)))
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	478 retry = self.retry
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	479 while True:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	480 try:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	481 self._basictransfer(obj, action, localstore)
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	482 yield 1, obj.get('oid')
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	483 break
35479 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	484 except socket.error as ex:
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	485 if retry > 0:
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	486 self.ui.note(
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	487 _('lfs: failed: %r (remaining retry %d)\n')
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	488 % (ex, retry))
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	489 retry -= 1
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	490 continue
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	491 raise
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	492
35732 10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	493 # Until https multiplexing gets sorted out
10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	494 if self.ui.configbool('experimental', 'lfs.worker-enable'):
10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	495 oids = worker.worker(self.ui, 0.1, transfer, (),
10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	496 sorted(objects, key=lambda o: o.get('oid')))
10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	497 else:
10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	498 oids = transfer(sorted(objects, key=lambda o: o.get('oid')))
10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	499
39416 b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	500 with self.ui.makeprogress(topic, total=total) as progress:
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	501 progress.update(0)
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	502 processed = 0
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	503 blobs = 0
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	504 for _one, oid in oids:
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	505 processed += sizes[oid]
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	506 blobs += 1
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	507 progress.update(processed)
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	508 self.ui.note(_('lfs: processed: %s\n') % oid)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	509
35921 fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	510 if blobs > 0:
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	511 if action == 'upload':
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	512 self.ui.status(_('lfs: uploaded %d files (%s)\n')
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	513 % (blobs, util.bytecount(processed)))
37765 ab04972a33ef lfs: enable the final download count status message Matt Harbison <matt_harbison@yahoo.com> parents: 37692 diff changeset	514 elif action == 'download':
ab04972a33ef lfs: enable the final download count status message Matt Harbison <matt_harbison@yahoo.com> parents: 37692 diff changeset	515 self.ui.status(_('lfs: downloaded %d files (%s)\n')
ab04972a33ef lfs: enable the final download count status message Matt Harbison <matt_harbison@yahoo.com> parents: 37692 diff changeset	516 % (blobs, util.bytecount(processed)))
35921 fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	517
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	518 def __del__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	519 # copied from mercurial/httppeer.py
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	520 urlopener = getattr(self, 'urlopener', None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	521 if urlopener:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	522 for h in urlopener.handlers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	523 h.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	524 getattr(h, "close_all", lambda : None)()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	525
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	526 class _dummyremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	527 """Dummy store storing blobs to temp directory."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	528
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	529 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	530 fullpath = repo.vfs.join('lfs', url.path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	531 self.vfs = lfsvfs(fullpath)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	532
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	533 def writebatch(self, pointers, fromstore):
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	534 for p in _deduplicate(pointers):
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	535 content = fromstore.read(p.oid(), verify=True)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	536 with self.vfs(p.oid(), 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	537 fp.write(content)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	538
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	539 def readbatch(self, pointers, tostore):
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	540 for p in _deduplicate(pointers):
35552 fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	541 with self.vfs(p.oid(), 'rb') as fp:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	542 tostore.download(p.oid(), fp)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	543
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	544 class _nullremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	545 """Null store storing blobs to /dev/null."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	546
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	547 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	548 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	549
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	550 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	551 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	552
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	553 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	554 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	555
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	556 class _promptremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	557 """Prompt user to set lfs.url when accessed."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	558
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	559 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	560 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	561
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	562 def writebatch(self, pointers, fromstore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	563 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	564
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	565 def readbatch(self, pointers, tostore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	566 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	567
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	568 def _prompt(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	569 raise error.Abort(_('lfs.url needs to be configured'))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	570
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	571 _storemap = {
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	572 'https': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	573 'http': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	574 'file': _dummyremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	575 'null': _nullremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	576 None: _promptremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	577 }
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	578
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	579 def _deduplicate(pointers):
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	580 """Remove any duplicate oids that exist in the list"""
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	581 reduced = util.sortdict()
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	582 for p in pointers:
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	583 reduced[p.oid()] = p
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	584 return reduced.values()
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	585
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	586 def _verify(oid, content):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	587 realoid = hashlib.sha256(content).hexdigest()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	588 if realoid != oid:
37692 10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	589 raise LfsCorruptionError(_('detected corrupt lfs object: %s') % oid,
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	590 hint=_('run hg verify'))
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	591
37564 31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push Matt Harbison <matt_harbison@yahoo.com> parents: 37563 diff changeset	592 def remote(repo, remote=None):
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	593 """remotestore factory. return a store in _storemap depending on config
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	594
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	595 If ``lfs.url`` is specified, use that remote endpoint. Otherwise, try to
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	596 infer the endpoint, based on the remote repository using the same path
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	597 adjustments as git. As an extension, 'http' is supported as well so that
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	598 ``hg serve`` works out of the box.
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	599
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	600 https://github.com/git-lfs/git-lfs/blob/master/docs/api/server-discovery.md
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	601 """
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	602 lfsurl = repo.ui.config('lfs', 'url')
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	603 url = util.url(lfsurl or '')
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	604 if lfsurl is None:
37564 31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push Matt Harbison <matt_harbison@yahoo.com> parents: 37563 diff changeset	605 if remote:
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	606 path = remote
37564 31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push Matt Harbison <matt_harbison@yahoo.com> parents: 37563 diff changeset	607 elif util.safehasattr(repo, '_subtoppath'):
37563 be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	608 # The pull command sets this during the optional update phase, which
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	609 # tells exactly where the pull originated, whether 'paths.default'
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	610 # or explicit.
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	611 path = repo._subtoppath
37563 be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	612 else:
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	613 # TODO: investigate 'paths.remote:lfsurl' style path customization,
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	614 # and fall back to inferring from 'paths.remote' if unspecified.
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	615 path = repo.ui.config('paths', 'default') or ''
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	616
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	617 defaulturl = util.url(path)
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	618
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	619 # TODO: support local paths as well.
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	620 # TODO: consider the ssh -> https transformation that git applies
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	621 if defaulturl.scheme in (b'http', b'https'):
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	622 if defaulturl.path and defaulturl.path[:-1] != b'/':
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	623 defaulturl.path += b'/'
37691 d241e6632669 lfs: fix the inferred remote store path when using a --prefix Matt Harbison <matt_harbison@yahoo.com> parents: 37565 diff changeset	624 defaulturl.path = (defaulturl.path or b'') + b'.git/info/lfs'
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	625
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	626 url = util.url(bytes(defaulturl))
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	627 repo.ui.note(_('lfs: assuming remote store: %s\n') % url)
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	628
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	629 scheme = url.scheme
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	630 if scheme not in _storemap:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	631 raise error.Abort(_('lfs: unknown url scheme: %s') % scheme)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	632 return _storemap[scheme](repo, url)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	633
39793 b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39474 diff changeset	634 class LfsRemoteError(error.StorageError):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	635 pass
37692 10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	636
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	637 class LfsCorruptionError(error.Abort):
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	638 """Raised when a corrupt blob is detected, aborting an operation
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	639
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	640 It exists to allow specialized handling on the server side."""

Mercurial > public > mercurial-scm > hg-stable

annotate hgext/lfs/blobstore.py @ 40703:93e5d18251d6