mercurial-scm/hg-stable: hgext/lfs/blobstore.py annotate

annotate hgext/lfs/blobstore.py @ 41453:0b636d1720a0

lfs: strip the response headers from the Batch API before printing For reasons unknown, py3 is adding an extra '\n' before the headers print out. This makes the output the same as py2.

author	Matt Harbison <matt_harbison@yahoo.com>
date	Sun, 27 Jan 2019 20:50:52 -0500
parents	7df10ea7a5b8
children	1bc01490178a

rev	line source
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	1 # blobstore.py - local and remote (speaking Git-LFS protocol) blob storages
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	2 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	3 # Copyright 2017 Facebook, Inc.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	4 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	7
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	8 from __future__ import absolute_import
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	9
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	10 import contextlib
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	11 import errno
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	12 import hashlib
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	13 import json
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	14 import os
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	15 import re
35479 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	16 import socket
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	17
35122 b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35121 diff changeset	18 from mercurial.i18n import _
b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35121 diff changeset	19
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	20 from mercurial import (
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	21 encoding,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	22 error,
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	23 node,
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	24 pathutil,
36630 4da09b46451e lfs: add some bytestring wrappers in blobstore.py Augie Fackler <augie@google.com> parents: 36485 diff changeset	25 pycompat,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	26 url as urlmod,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	27 util,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	28 vfs as vfsmod,
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	29 worker,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	30 )
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	31
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	32 from mercurial.utils import (
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	33 stringutil,
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	34 )
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	35
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	36 from ..largefiles import lfutil
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	37
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	38 # 64 bytes for SHA256
36485 9e3cb58c7ab3 py3: make sure regexes are bytes Pulkit Goyal <7895pulkit@gmail.com> parents: 35967 diff changeset	39 _lfsre = re.compile(br'\A[a-f0-9]{64}\Z')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	40
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	41 class lfsvfs(vfsmod.vfs):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	42 def join(self, path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	43 """split the path at first two characters, like: XX/XXXXX..."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	44 if not _lfsre.match(path):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	45 raise error.ProgrammingError(b'unexpected lfs path: %s' % path)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	46 return super(lfsvfs, self).join(path[0:2], path[2:])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	47
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	48 def walk(self, path=None, onerror=None):
35403 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35371 diff changeset	49 """Yield (dirpath, [], oids) tuple for blobs under path
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	50
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	51 Oids only exist in the root of this vfs, so dirpath is always ''.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	52 """
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	53 root = os.path.normpath(self.base)
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	54 # when dirpath == root, dirpath[prefixlen:] becomes empty
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	55 # because len(dirpath) < prefixlen.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	56 prefixlen = len(pathutil.normasprefix(root))
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	57 oids = []
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	58
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	59 for dirpath, dirs, files in os.walk(self.reljoin(self.base, path
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	60 or b''),
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	61 onerror=onerror):
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	62 dirpath = dirpath[prefixlen:]
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	63
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	64 # Silently skip unexpected files and directories
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	65 if len(dirpath) == 2:
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	66 oids.extend([dirpath + f for f in files
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	67 if _lfsre.match(dirpath + f)])
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	68
35403 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35371 diff changeset	69 yield ('', [], oids)
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	70
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	71 class nullvfs(lfsvfs):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	72 def __init__(self):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	73 pass
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	74
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	75 def exists(self, oid):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	76 return False
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	77
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	78 def read(self, oid):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	79 # store.read() calls into here if the blob doesn't exist in its
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	80 # self.vfs. Raise the same error as a normal vfs when asked to read a
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	81 # file that doesn't exist. The only difference is the full file path
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	82 # isn't available in the error.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	83 raise IOError(errno.ENOENT,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	84 pycompat.sysstr(b'%s: No such file or directory' % oid))
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	85
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	86 def walk(self, path=None, onerror=None):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	87 return (b'', [], [])
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	88
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	89 def write(self, oid, data):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	90 pass
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	91
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	92 class filewithprogress(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	93 """a file-like object that supports __len__ and read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	94
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	95 Useful to provide progress information for how many bytes are read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	96 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	97
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	98 def __init__(self, fp, callback):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	99 self._fp = fp
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	100 self._callback = callback # func(readsize)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	101 fp.seek(0, os.SEEK_END)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	102 self._len = fp.tell()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	103 fp.seek(0)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	104
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	105 def __len__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	106 return self._len
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	107
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	108 def read(self, size):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	109 if self._fp is None:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	110 return b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	111 data = self._fp.read(size)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	112 if data:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	113 if self._callback:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	114 self._callback(len(data))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	115 else:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	116 self._fp.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	117 self._fp = None
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	118 return data
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	119
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	120 class local(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	121 """Local blobstore for large file contents.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	122
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	123 This blobstore is used both as a cache and as a staging area for large blobs
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	124 to be uploaded to the remote blobstore.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	125 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	126
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	127 def __init__(self, repo):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	128 fullpath = repo.svfs.join(b'lfs/objects')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	129 self.vfs = lfsvfs(fullpath)
37562 e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url Matt Harbison <matt_harbison@yahoo.com> parents: 37518 diff changeset	130
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	131 if repo.ui.configbool(b'experimental', b'lfs.disableusercache'):
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	132 self.cachevfs = nullvfs()
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	133 else:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	134 usercache = lfutil._usercachedir(repo.ui, b'lfs')
37562 e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url Matt Harbison <matt_harbison@yahoo.com> parents: 37518 diff changeset	135 self.cachevfs = lfsvfs(usercache)
35477 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35444 diff changeset	136 self.ui = repo.ui
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	137
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	138 def open(self, oid):
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	139 """Open a read-only file descriptor to the named blob, in either the
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	140 usercache or the local store."""
35541 58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	141 # The usercache is the most likely place to hold the file. Commit will
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	142 # write to both it and the local store, as will anything that downloads
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	143 # the blobs. However, things like clone without an update won't
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	144 # populate the local store. For an init + push of a local clone,
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	145 # the usercache is the only place it _could_ be. If not present, the
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	146 # missing file msg here will indicate the local repo, not the usercache.
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	147 if self.cachevfs.exists(oid):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	148 return self.cachevfs(oid, b'rb')
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	149
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	150 return self.vfs(oid, b'rb')
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	151
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	152 def download(self, oid, src):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	153 """Read the blob from the remote source in chunks, verify the content,
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	154 and write to this local blobstore."""
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	155 sha256 = hashlib.sha256()
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	156
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	157 with self.vfs(oid, b'wb', atomictemp=True) as fp:
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	158 for chunk in util.filechunkiter(src, size=1048576):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	159 fp.write(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	160 sha256.update(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	161
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	162 realoid = node.hex(sha256.digest())
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	163 if realoid != oid:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	164 raise LfsCorruptionError(_(b'corrupt remote lfs object: %s')
37692 10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	165 % oid)
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	166
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	167 self._linktousercache(oid)
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	168
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	169 def write(self, oid, data):
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	170 """Write blob to local blobstore.
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	171
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	172 This should only be called from the filelog during a commit or similar.
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	173 As such, there is no need to verify the data. Imports from a remote
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	174 store must use ``download()`` instead."""
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	175 with self.vfs(oid, b'wb', atomictemp=True) as fp:
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	176 fp.write(data)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	177
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	178 self._linktousercache(oid)
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	179
39474 a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	180 def linkfromusercache(self, oid):
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	181 """Link blobs found in the user cache into this store.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	182
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	183 The server module needs to do this when it lets the client know not to
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	184 upload the blob, to ensure it is always available in this store.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	185 Normally this is done implicitly when the client reads or writes the
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	186 blob, but that doesn't happen when the server tells the client that it
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	187 already has the blob.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	188 """
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	189 if (not isinstance(self.cachevfs, nullvfs)
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	190 and not self.vfs.exists(oid)):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	191 self.ui.note(_(b'lfs: found %s in the usercache\n') % oid)
39474 a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	192 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	193
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	194 def _linktousercache(self, oid):
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	195 # XXX: should we verify the content of the cache, and hardlink back to
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	196 # the local store on success, but truncate, write and link on failure?
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	197 if (not self.cachevfs.exists(oid)
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	198 and not isinstance(self.cachevfs, nullvfs)):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	199 self.ui.note(_(b'lfs: adding %s to the usercache\n') % oid)
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	200 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	201
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	202 def read(self, oid, verify=True):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	203 """Read blob from local blobstore."""
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	204 if not self.vfs.exists(oid):
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	205 blob = self._read(self.cachevfs, oid, verify)
35481 bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	206
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	207 # Even if revlog will verify the content, it needs to be verified
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	208 # now before making the hardlink to avoid propagating corrupt blobs.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	209 # Don't abort if corruption is detected, because `hg verify` will
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	210 # give more useful info about the corruption- simply don't add the
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	211 # hardlink.
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	212 if verify or node.hex(hashlib.sha256(blob).digest()) == oid:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	213 self.ui.note(_(b'lfs: found %s in the usercache\n') % oid)
35481 bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	214 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
35477 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35444 diff changeset	215 else:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	216 self.ui.note(_(b'lfs: found %s in the local lfs store\n') % oid)
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	217 blob = self._read(self.vfs, oid, verify)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	218 return blob
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	219
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	220 def _read(self, vfs, oid, verify):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	221 """Read blob (after verifying) from the given store"""
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	222 blob = vfs.read(oid)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	223 if verify:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	224 _verify(oid, blob)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	225 return blob
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	226
37148 56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	227 def verify(self, oid):
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	228 """Indicate whether or not the hash of the underlying file matches its
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	229 name."""
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	230 sha256 = hashlib.sha256()
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	231
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	232 with self.open(oid) as fp:
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	233 for chunk in util.filechunkiter(fp, size=1048576):
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	234 sha256.update(chunk)
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	235
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	236 return oid == node.hex(sha256.digest())
37148 56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	237
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	238 def has(self, oid):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	239 """Returns True if the local blobstore contains the requested blob,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	240 False otherwise."""
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	241 return self.cachevfs.exists(oid) or self.vfs.exists(oid)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	242
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	243 def _urlerrorreason(urlerror):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	244 '''Create a friendly message for the given URLError to be used in an
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	245 LfsRemoteError message.
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	246 '''
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	247 inst = urlerror
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	248
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	249 if isinstance(urlerror.reason, Exception):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	250 inst = urlerror.reason
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	251
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	252 if util.safehasattr(inst, 'reason'):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	253 try: # usually it is in the form (errno, strerror)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	254 reason = inst.reason.args[1]
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	255 except (AttributeError, IndexError):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	256 # it might be anything, for example a string
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	257 reason = inst.reason
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	258 if isinstance(reason, pycompat.unicode):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	259 # SSLError of Python 2.7.9 contains a unicode
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	260 reason = encoding.unitolocal(reason)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	261 return reason
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	262 elif getattr(inst, "strerror", None):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	263 return encoding.strtolocal(inst.strerror)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	264 else:
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	265 return stringutil.forcebytestr(urlerror)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	266
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	267 class _gitlfsremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	268
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	269 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	270 ui = repo.ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	271 self.ui = ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	272 baseurl, authinfo = url.authinfo()
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	273 self.baseurl = baseurl.rstrip(b'/')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	274 useragent = repo.ui.config(b'experimental', b'lfs.user-agent')
35444 e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35443 diff changeset	275 if not useragent:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	276 useragent = b'git-lfs/2.3.4 (Mercurial %s)' % util.version()
35443 e7bb5fc4570c lfs: add git to the User-Agent header for blob transfers Matt Harbison <matt_harbison@yahoo.com> parents: 35437 diff changeset	277 self.urlopener = urlmod.opener(ui, authinfo, useragent)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	278 self.retry = ui.configint(b'lfs', b'retry')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	279
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	280 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	281 """Batch upload from local to remote blobstore."""
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	282 self._batch(_deduplicate(pointers), fromstore, b'upload')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	283
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	284 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	285 """Batch download from remote to local blostore."""
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	286 self._batch(_deduplicate(pointers), tostore, b'download')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	287
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	288 def _batchrequest(self, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	289 """Get metadata about objects pointed by pointers for given action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	290
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	291 Return decoded JSON object like {'objects': [{'oid': '', 'size': 1}]}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	292 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	293 """
41449 40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	294 objects = [{r'oid': pycompat.strurl(p.oid()),
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	295 r'size': p.size()} for p in pointers]
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	296 requestdata = pycompat.bytesurl(json.dumps({
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	297 r'objects': objects,
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	298 r'operation': pycompat.strurl(action),
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	299 }))
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	300 url = b'%s/objects/batch' % self.baseurl
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	301 batchreq = util.urlreq.request(pycompat.strurl(url), data=requestdata)
41449 40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	302 batchreq.add_header(r'Accept', r'application/vnd.git-lfs+json')
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	303 batchreq.add_header(r'Content-Type', r'application/vnd.git-lfs+json')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	304 try:
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	305 with contextlib.closing(self.urlopener.open(batchreq)) as rsp:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	306 rawjson = rsp.read()
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	307 except util.urlerr.httperror as ex:
40701 9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	308 hints = {
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	309 400: _(b'check that lfs serving is enabled on %s and "%s" is '
41451 7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	310 b'supported') % (self.baseurl, action),
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	311 404: _(b'the "lfs.url" config may be used to override %s')
40701 9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	312 % self.baseurl,
9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	313 }
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	314 hint = hints.get(ex.code, _(b'api=%s, action=%s') % (url, action))
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	315 raise LfsRemoteError(
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	316 _(b'LFS HTTP error: %s') % stringutil.forcebytestr(ex),
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	317 hint=hint)
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	318 except util.urlerr.urlerror as ex:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	319 hint = (_(b'the "lfs.url" config may be used to override %s')
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	320 % self.baseurl)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	321 raise LfsRemoteError(_(b'LFS error: %s') % _urlerrorreason(ex),
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	322 hint=hint)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	323 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	324 response = json.loads(rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	325 except ValueError:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	326 raise LfsRemoteError(_(b'LFS server returns invalid JSON: %s')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	327 % rawjson.encode("utf-8"))
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	328
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	329 if self.ui.debugflag:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	330 self.ui.debug(b'Status: %d\n' % rsp.status)
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	331 # lfs-test-server and hg serve return headers in different order
41453 0b636d1720a0 lfs: strip the response headers from the Batch API before printing Matt Harbison <matt_harbison@yahoo.com> parents: 41451 diff changeset	332 headers = pycompat.bytestr(rsp.info()).strip()
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	333 self.ui.debug(b'%s\n'
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	334 % b'\n'.join(sorted(headers.splitlines())))
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	335
41449 40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	336 if r'objects' in response:
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	337 response[r'objects'] = sorted(response[r'objects'],
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	338 key=lambda p: p[r'oid'])
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	339 self.ui.debug(b'%s\n'
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	340 % pycompat.bytesurl(
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	341 json.dumps(response, indent=2,
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	342 separators=(r'', r': '),
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	343 sort_keys=True)))
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	344
41451 7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	345 def encodestr(x):
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	346 if isinstance(x, pycompat.unicode):
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	347 return x.encode(u'utf-8')
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	348 return x
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	349
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	350 return pycompat.rapply(encodestr, response)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	351
35666 2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	352 def _checkforservererror(self, pointers, responses, action):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	353 """Scans errors from objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	354
35694 8a23082f4d93 lfs: correct documentation typo Matt Harbison <matt_harbison@yahoo.com> parents: 35666 diff changeset	355 Raises LfsRemoteError if any objects have an error"""
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	356 for response in responses:
35666 2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	357 # The server should return 404 when objects cannot be found. Some
2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	358 # server implementation (ex. lfs-test-server) does not set "error"
2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	359 # but just removes "download" from "actions". Treat that case
2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	360 # as the same as 404 error.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	361 if b'error' not in response:
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	362 if (action == b'download'
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	363 and action not in response.get(b'actions', [])):
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	364 code = 404
35695 dd672e3d059f lfs: raise an error if the server sends an unsolicited oid Matt Harbison <matt_harbison@yahoo.com> parents: 35694 diff changeset	365 else:
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	366 continue
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	367 else:
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	368 # An error dict without a code doesn't make much sense, so
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	369 # treat as a server error.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	370 code = response.get(b'error').get(b'code', 500)
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	371
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	372 ptrmap = {p.oid(): p for p in pointers}
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	373 p = ptrmap.get(response[b'oid'], None)
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	374 if p:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	375 filename = getattr(p, 'filename', b'unknown')
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	376 errors = {
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	377 404: b'The object does not exist',
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	378 410: b'The object was removed by the owner',
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	379 422: b'Validation error',
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	380 500: b'Internal server error',
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	381 }
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	382 msg = errors.get(code, b'status code %d' % code)
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	383 raise LfsRemoteError(_(b'LFS server error for "%s": %s')
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	384 % (filename, msg))
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	385 else:
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	386 raise LfsRemoteError(
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	387 _(b'LFS server error. Unsolicited response for oid %s')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	388 % response[b'oid'])
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	389
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	390 def _extractobjects(self, response, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	391 """extract objects from response of the batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	392
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	393 response: parsed JSON object returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	394 return response['objects'] filtered by action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	395 raise if any object has an error
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	396 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	397 # Scan errors from objects - fail early
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	398 objects = response.get(b'objects', [])
35666 2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	399 self._checkforservererror(pointers, objects, action)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	400
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	401 # Filter objects with given action. Practically, this skips uploading
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	402 # objects which exist in the server.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	403 filteredobjects = [o for o in objects
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	404 if action in o.get(b'actions', [])]
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	405
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	406 return filteredobjects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	407
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	408 def _basictransfer(self, obj, action, localstore):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	409 """Download or upload a single object using basic transfer protocol
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	410
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	411 obj: dict, an object description returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	412 action: string, one of ['upload', 'download']
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	413 localstore: blobstore.local
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	414
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	415 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/\
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	416 basic-transfers.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	417 """
41451 7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	418 oid = obj[b'oid']
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	419 href = obj[b'actions'][action].get(b'href')
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	420 headers = obj[b'actions'][action].get(b'header', {}).items()
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	421
41451 7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	422 request = util.urlreq.request(pycompat.strurl(href))
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	423 if action == b'upload':
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	424 # If uploading blobs, read data from local blobstore.
37219 b00bd974eef5 lfs: drop a duplicate blob verification method Matt Harbison <matt_harbison@yahoo.com> parents: 37149 diff changeset	425 if not localstore.verify(oid):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	426 raise error.Abort(_(b'detected corrupt lfs object: %s') % oid,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	427 hint=_(b'run hg verify'))
35530 e8f80529abeb lfs: use the local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35529 diff changeset	428 request.data = filewithprogress(localstore.open(oid), None)
41449 40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	429 request.get_method = lambda: r'PUT'
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	430 request.add_header(r'Content-Type', r'application/octet-stream')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	431
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	432 for k, v in headers:
41451 7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore Matt Harbison <matt_harbison@yahoo.com> parents: 41449 diff changeset	433 request.add_header(pycompat.strurl(k), pycompat.strurl(v))
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	434
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	435 response = b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	436 try:
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	437 with contextlib.closing(self.urlopener.open(request)) as req:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	438 ui = self.ui # Shorten debug lines
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	439 if self.ui.debugflag:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	440 ui.debug(b'Status: %d\n' % req.status)
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	441 # lfs-test-server and hg serve return headers in different
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	442 # order
41453 0b636d1720a0 lfs: strip the response headers from the Batch API before printing Matt Harbison <matt_harbison@yahoo.com> parents: 41451 diff changeset	443 headers = pycompat.bytestr(req.info()).strip()
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	444 ui.debug(b'%s\n'
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	445 % b'\n'.join(sorted(headers.splitlines())))
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	446
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	447 if action == b'download':
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	448 # If downloading blobs, store downloaded data to local
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	449 # blobstore
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	450 localstore.download(oid, req)
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	451 else:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	452 while True:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	453 data = req.read(1048576)
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	454 if not data:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	455 break
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	456 response += data
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	457 if response:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	458 ui.debug(b'lfs %s response: %s' % (action, response))
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	459 except util.urlerr.httperror as ex:
35734 b4e1d0654736 lfs: dump the full response on httperror in debug mode Matt Harbison <matt_harbison@yahoo.com> parents: 35733 diff changeset	460 if self.ui.debugflag:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	461 self.ui.debug(b'%s: %s\n' % (oid, ex.read())) # XXX: also bytes?
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	462 raise LfsRemoteError(_(b'LFS HTTP error: %s (oid=%s, action=%s)')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	463 % (stringutil.forcebytestr(ex), oid, action))
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	464 except util.urlerr.urlerror as ex:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	465 hint = (_(b'attempted connection to %s')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	466 % pycompat.bytesurl(util.urllibcompat.getfullurl(request)))
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	467 raise LfsRemoteError(_(b'LFS error: %s') % _urlerrorreason(ex),
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	468 hint=hint)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	469
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	470 def _batch(self, pointers, localstore, action):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	471 if action not in [b'upload', b'download']:
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	472 raise error.ProgrammingError(b'invalid Git-LFS action: %s' % action)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	473
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	474 response = self._batchrequest(pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	475 objects = self._extractobjects(response, pointers, action)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	476 total = sum(x.get(b'size', 0) for x in objects)
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	477 sizes = {}
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	478 for obj in objects:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	479 sizes[obj.get(b'oid')] = obj.get(b'size', 0)
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	480 topic = {b'upload': _(b'lfs uploading'),
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	481 b'download': _(b'lfs downloading')}[action]
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	482 if len(objects) > 1:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	483 self.ui.note(_(b'lfs: need to transfer %d objects (%s)\n')
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	484 % (len(objects), util.bytecount(total)))
39416 b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	485
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	486 def transfer(chunk):
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	487 for obj in chunk:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	488 objsize = obj.get(b'size', 0)
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	489 if self.ui.verbose:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	490 if action == b'download':
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	491 msg = _(b'lfs: downloading %s (%s)\n')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	492 elif action == b'upload':
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	493 msg = _(b'lfs: uploading %s (%s)\n')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	494 self.ui.note(msg % (obj.get(b'oid'),
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	495 util.bytecount(objsize)))
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	496 retry = self.retry
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	497 while True:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	498 try:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	499 self._basictransfer(obj, action, localstore)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	500 yield 1, obj.get(b'oid')
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	501 break
35479 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	502 except socket.error as ex:
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	503 if retry > 0:
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	504 self.ui.note(
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	505 _(b'lfs: failed: %r (remaining retry %d)\n')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	506 % (stringutil.forcebytestr(ex), retry))
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	507 retry -= 1
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	508 continue
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	509 raise
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	510
35732 10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	511 # Until https multiplexing gets sorted out
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	512 if self.ui.configbool(b'experimental', b'lfs.worker-enable'):
35732 10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	513 oids = worker.worker(self.ui, 0.1, transfer, (),
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	514 sorted(objects, key=lambda o: o.get(b'oid')))
35732 10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	515 else:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	516 oids = transfer(sorted(objects, key=lambda o: o.get(b'oid')))
35732 10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	517
39416 b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	518 with self.ui.makeprogress(topic, total=total) as progress:
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	519 progress.update(0)
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	520 processed = 0
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	521 blobs = 0
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	522 for _one, oid in oids:
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	523 processed += sizes[oid]
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	524 blobs += 1
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	525 progress.update(processed)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	526 self.ui.note(_(b'lfs: processed: %s\n') % oid)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	527
35921 fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	528 if blobs > 0:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	529 if action == b'upload':
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	530 self.ui.status(_(b'lfs: uploaded %d files (%s)\n')
35921 fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	531 % (blobs, util.bytecount(processed)))
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	532 elif action == b'download':
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	533 self.ui.status(_(b'lfs: downloaded %d files (%s)\n')
37765 ab04972a33ef lfs: enable the final download count status message Matt Harbison <matt_harbison@yahoo.com> parents: 37692 diff changeset	534 % (blobs, util.bytecount(processed)))
35921 fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	535
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	536 def __del__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	537 # copied from mercurial/httppeer.py
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	538 urlopener = getattr(self, 'urlopener', None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	539 if urlopener:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	540 for h in urlopener.handlers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	541 h.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	542 getattr(h, "close_all", lambda : None)()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	543
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	544 class _dummyremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	545 """Dummy store storing blobs to temp directory."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	546
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	547 def __init__(self, repo, url):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	548 fullpath = repo.vfs.join(b'lfs', url.path)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	549 self.vfs = lfsvfs(fullpath)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	550
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	551 def writebatch(self, pointers, fromstore):
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	552 for p in _deduplicate(pointers):
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	553 content = fromstore.read(p.oid(), verify=True)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	554 with self.vfs(p.oid(), b'wb', atomictemp=True) as fp:
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	555 fp.write(content)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	556
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	557 def readbatch(self, pointers, tostore):
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	558 for p in _deduplicate(pointers):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	559 with self.vfs(p.oid(), b'rb') as fp:
35552 fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	560 tostore.download(p.oid(), fp)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	561
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	562 class _nullremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	563 """Null store storing blobs to /dev/null."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	564
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	565 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	566 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	567
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	568 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	569 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	570
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	571 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	572 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	573
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	574 class _promptremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	575 """Prompt user to set lfs.url when accessed."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	576
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	577 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	578 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	579
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	580 def writebatch(self, pointers, fromstore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	581 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	582
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	583 def readbatch(self, pointers, tostore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	584 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	585
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	586 def _prompt(self):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	587 raise error.Abort(_(b'lfs.url needs to be configured'))
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	588
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	589 _storemap = {
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	590 b'https': _gitlfsremote,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	591 b'http': _gitlfsremote,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	592 b'file': _dummyremote,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	593 b'null': _nullremote,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	594 None: _promptremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	595 }
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	596
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	597 def _deduplicate(pointers):
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	598 """Remove any duplicate oids that exist in the list"""
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	599 reduced = util.sortdict()
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	600 for p in pointers:
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	601 reduced[p.oid()] = p
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	602 return reduced.values()
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	603
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	604 def _verify(oid, content):
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	605 realoid = node.hex(hashlib.sha256(content).digest())
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	606 if realoid != oid:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	607 raise LfsCorruptionError(_(b'detected corrupt lfs object: %s') % oid,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	608 hint=_(b'run hg verify'))
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	609
37564 31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push Matt Harbison <matt_harbison@yahoo.com> parents: 37563 diff changeset	610 def remote(repo, remote=None):
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	611 """remotestore factory. return a store in _storemap depending on config
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	612
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	613 If ``lfs.url`` is specified, use that remote endpoint. Otherwise, try to
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	614 infer the endpoint, based on the remote repository using the same path
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	615 adjustments as git. As an extension, 'http' is supported as well so that
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	616 ``hg serve`` works out of the box.
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	617
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	618 https://github.com/git-lfs/git-lfs/blob/master/docs/api/server-discovery.md
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	619 """
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	620 lfsurl = repo.ui.config(b'lfs', b'url')
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	621 url = util.url(lfsurl or '')
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	622 if lfsurl is None:
37564 31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push Matt Harbison <matt_harbison@yahoo.com> parents: 37563 diff changeset	623 if remote:
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	624 path = remote
37564 31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push Matt Harbison <matt_harbison@yahoo.com> parents: 37563 diff changeset	625 elif util.safehasattr(repo, '_subtoppath'):
37563 be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	626 # The pull command sets this during the optional update phase, which
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	627 # tells exactly where the pull originated, whether 'paths.default'
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	628 # or explicit.
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	629 path = repo._subtoppath
37563 be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	630 else:
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	631 # TODO: investigate 'paths.remote:lfsurl' style path customization,
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	632 # and fall back to inferring from 'paths.remote' if unspecified.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	633 path = repo.ui.config(b'paths', b'default') or b''
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	634
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	635 defaulturl = util.url(path)
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	636
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	637 # TODO: support local paths as well.
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	638 # TODO: consider the ssh -> https transformation that git applies
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	639 if defaulturl.scheme in (b'http', b'https'):
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	640 if defaulturl.path and defaulturl.path[:-1] != b'/':
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	641 defaulturl.path += b'/'
37691 d241e6632669 lfs: fix the inferred remote store path when using a --prefix Matt Harbison <matt_harbison@yahoo.com> parents: 37565 diff changeset	642 defaulturl.path = (defaulturl.path or b'') + b'.git/info/lfs'
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	643
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	644 url = util.url(bytes(defaulturl))
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	645 repo.ui.note(_(b'lfs: assuming remote store: %s\n') % url)
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	646
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	647 scheme = url.scheme
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	648 if scheme not in _storemap:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	649 raise error.Abort(_(b'lfs: unknown url scheme: %s') % scheme)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	650 return _storemap[scheme](repo, url)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	651
39793 b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39474 diff changeset	652 class LfsRemoteError(error.StorageError):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	653 pass
37692 10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	654
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	655 class LfsCorruptionError(error.Abort):
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	656 """Raised when a corrupt blob is detected, aborting an operation
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	657
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	658 It exists to allow specialized handling on the server side."""

Mercurial > public > mercurial-scm > hg-stable

annotate hgext/lfs/blobstore.py @ 41453:0b636d1720a0