mercurial-scm/hg-stable: hgext/lfs/blobstore.py annotate

annotate hgext/lfs/blobstore.py @ 41449:40efcf78f3df

py3: raw stringify various JSON and HTTP headers in the LFS blobstore module This is (almost?) entirely from Augie's work. I'm a bit surprised that the JSON data is being encoded with ASCII via `pycompat.bytesurl()`- I would have thought UTF-8.

author	Matt Harbison <matt_harbison@yahoo.com>
date	Sun, 27 Jan 2019 15:42:55 -0500
parents	02d0a7774882
children	7df10ea7a5b8

rev	line source
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	1 # blobstore.py - local and remote (speaking Git-LFS protocol) blob storages
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	2 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	3 # Copyright 2017 Facebook, Inc.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	4 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	7
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	8 from __future__ import absolute_import
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	9
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	10 import contextlib
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	11 import errno
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	12 import hashlib
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	13 import json
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	14 import os
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	15 import re
35479 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	16 import socket
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	17
35122 b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35121 diff changeset	18 from mercurial.i18n import _
b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35121 diff changeset	19
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	20 from mercurial import (
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	21 encoding,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	22 error,
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	23 node,
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	24 pathutil,
36630 4da09b46451e lfs: add some bytestring wrappers in blobstore.py Augie Fackler <augie@google.com> parents: 36485 diff changeset	25 pycompat,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	26 url as urlmod,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	27 util,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	28 vfs as vfsmod,
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	29 worker,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	30 )
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	31
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	32 from mercurial.utils import (
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	33 stringutil,
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	34 )
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	35
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	36 from ..largefiles import lfutil
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	37
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	38 # 64 bytes for SHA256
36485 9e3cb58c7ab3 py3: make sure regexes are bytes Pulkit Goyal <7895pulkit@gmail.com> parents: 35967 diff changeset	39 _lfsre = re.compile(br'\A[a-f0-9]{64}\Z')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	40
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	41 class lfsvfs(vfsmod.vfs):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	42 def join(self, path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	43 """split the path at first two characters, like: XX/XXXXX..."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	44 if not _lfsre.match(path):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	45 raise error.ProgrammingError(b'unexpected lfs path: %s' % path)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	46 return super(lfsvfs, self).join(path[0:2], path[2:])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	47
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	48 def walk(self, path=None, onerror=None):
35403 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35371 diff changeset	49 """Yield (dirpath, [], oids) tuple for blobs under path
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	50
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	51 Oids only exist in the root of this vfs, so dirpath is always ''.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	52 """
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	53 root = os.path.normpath(self.base)
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	54 # when dirpath == root, dirpath[prefixlen:] becomes empty
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	55 # because len(dirpath) < prefixlen.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	56 prefixlen = len(pathutil.normasprefix(root))
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	57 oids = []
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	58
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	59 for dirpath, dirs, files in os.walk(self.reljoin(self.base, path
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	60 or b''),
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	61 onerror=onerror):
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	62 dirpath = dirpath[prefixlen:]
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	63
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	64 # Silently skip unexpected files and directories
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	65 if len(dirpath) == 2:
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	66 oids.extend([dirpath + f for f in files
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	67 if _lfsre.match(dirpath + f)])
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	68
35403 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35371 diff changeset	69 yield ('', [], oids)
35371 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35289 diff changeset	70
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	71 class nullvfs(lfsvfs):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	72 def __init__(self):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	73 pass
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	74
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	75 def exists(self, oid):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	76 return False
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	77
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	78 def read(self, oid):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	79 # store.read() calls into here if the blob doesn't exist in its
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	80 # self.vfs. Raise the same error as a normal vfs when asked to read a
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	81 # file that doesn't exist. The only difference is the full file path
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	82 # isn't available in the error.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	83 raise IOError(errno.ENOENT,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	84 pycompat.sysstr(b'%s: No such file or directory' % oid))
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	85
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	86 def walk(self, path=None, onerror=None):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	87 return (b'', [], [])
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	88
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	89 def write(self, oid, data):
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	90 pass
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	91
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	92 class filewithprogress(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	93 """a file-like object that supports __len__ and read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	94
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	95 Useful to provide progress information for how many bytes are read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	96 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	97
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	98 def __init__(self, fp, callback):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	99 self._fp = fp
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	100 self._callback = callback # func(readsize)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	101 fp.seek(0, os.SEEK_END)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	102 self._len = fp.tell()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	103 fp.seek(0)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	104
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	105 def __len__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	106 return self._len
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	107
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	108 def read(self, size):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	109 if self._fp is None:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	110 return b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	111 data = self._fp.read(size)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	112 if data:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	113 if self._callback:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	114 self._callback(len(data))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	115 else:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	116 self._fp.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	117 self._fp = None
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	118 return data
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	119
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	120 class local(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	121 """Local blobstore for large file contents.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	122
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	123 This blobstore is used both as a cache and as a staging area for large blobs
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	124 to be uploaded to the remote blobstore.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	125 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	126
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	127 def __init__(self, repo):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	128 fullpath = repo.svfs.join(b'lfs/objects')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	129 self.vfs = lfsvfs(fullpath)
37562 e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url Matt Harbison <matt_harbison@yahoo.com> parents: 37518 diff changeset	130
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	131 if repo.ui.configbool(b'experimental', b'lfs.disableusercache'):
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	132 self.cachevfs = nullvfs()
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	133 else:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	134 usercache = lfutil._usercachedir(repo.ui, b'lfs')
37562 e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url Matt Harbison <matt_harbison@yahoo.com> parents: 37518 diff changeset	135 self.cachevfs = lfsvfs(usercache)
35477 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35444 diff changeset	136 self.ui = repo.ui
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	137
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	138 def open(self, oid):
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	139 """Open a read-only file descriptor to the named blob, in either the
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	140 usercache or the local store."""
35541 58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	141 # The usercache is the most likely place to hold the file. Commit will
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	142 # write to both it and the local store, as will anything that downloads
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	143 # the blobs. However, things like clone without an update won't
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	144 # populate the local store. For an init + push of a local clone,
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	145 # the usercache is the only place it _could_ be. If not present, the
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35530 diff changeset	146 # missing file msg here will indicate the local repo, not the usercache.
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	147 if self.cachevfs.exists(oid):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	148 return self.cachevfs(oid, b'rb')
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	149
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	150 return self.vfs(oid, b'rb')
35529 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35482 diff changeset	151
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	152 def download(self, oid, src):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	153 """Read the blob from the remote source in chunks, verify the content,
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	154 and write to this local blobstore."""
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	155 sha256 = hashlib.sha256()
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	156
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	157 with self.vfs(oid, b'wb', atomictemp=True) as fp:
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	158 for chunk in util.filechunkiter(src, size=1048576):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	159 fp.write(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	160 sha256.update(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	161
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	162 realoid = node.hex(sha256.digest())
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	163 if realoid != oid:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	164 raise LfsCorruptionError(_(b'corrupt remote lfs object: %s')
37692 10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	165 % oid)
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	166
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	167 self._linktousercache(oid)
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35541 diff changeset	168
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	169 def write(self, oid, data):
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	170 """Write blob to local blobstore.
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	171
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	172 This should only be called from the filelog during a commit or similar.
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	173 As such, there is no need to verify the data. Imports from a remote
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	174 store must use ``download()`` instead."""
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	175 with self.vfs(oid, b'wb', atomictemp=True) as fp:
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	176 fp.write(data)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	177
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	178 self._linktousercache(oid)
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	179
39474 a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	180 def linkfromusercache(self, oid):
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	181 """Link blobs found in the user cache into this store.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	182
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	183 The server module needs to do this when it lets the client know not to
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	184 upload the blob, to ensure it is always available in this store.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	185 Normally this is done implicitly when the client reads or writes the
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	186 blob, but that doesn't happen when the server tells the client that it
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	187 already has the blob.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	188 """
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	189 if (not isinstance(self.cachevfs, nullvfs)
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	190 and not self.vfs.exists(oid)):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	191 self.ui.note(_(b'lfs: found %s in the usercache\n') % oid)
39474 a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	192 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads Matt Harbison <matt_harbison@yahoo.com> parents: 39416 diff changeset	193
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	194 def _linktousercache(self, oid):
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	195 # XXX: should we verify the content of the cache, and hardlink back to
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	196 # the local store on success, but truncate, write and link on failure?
37517 491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	197 if (not self.cachevfs.exists(oid)
491edf2435a0 lfs: add the ability to disable the usercache Matt Harbison <matt_harbison@yahoo.com> parents: 37245 diff changeset	198 and not isinstance(self.cachevfs, nullvfs)):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	199 self.ui.note(_(b'lfs: adding %s to the usercache\n') % oid)
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	200 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	201
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	202 def read(self, oid, verify=True):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	203 """Read blob from local blobstore."""
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	204 if not self.vfs.exists(oid):
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	205 blob = self._read(self.cachevfs, oid, verify)
35481 bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	206
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	207 # Even if revlog will verify the content, it needs to be verified
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	208 # now before making the hardlink to avoid propagating corrupt blobs.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	209 # Don't abort if corruption is detected, because `hg verify` will
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	210 # give more useful info about the corruption- simply don't add the
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	211 # hardlink.
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	212 if verify or node.hex(hashlib.sha256(blob).digest()) == oid:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	213 self.ui.note(_(b'lfs: found %s in the usercache\n') % oid)
35481 bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35480 diff changeset	214 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
35477 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35444 diff changeset	215 else:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	216 self.ui.note(_(b'lfs: found %s in the local lfs store\n') % oid)
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	217 blob = self._read(self.vfs, oid, verify)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	218 return blob
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	219
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	220 def _read(self, vfs, oid, verify):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	221 """Read blob (after verifying) from the given store"""
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	222 blob = vfs.read(oid)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	223 if verify:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	224 _verify(oid, blob)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	225 return blob
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	226
37148 56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	227 def verify(self, oid):
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	228 """Indicate whether or not the hash of the underlying file matches its
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	229 name."""
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	230 sha256 = hashlib.sha256()
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	231
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	232 with self.open(oid) as fp:
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	233 for chunk in util.filechunkiter(fp, size=1048576):
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	234 sha256.update(chunk)
56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	235
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	236 return oid == node.hex(sha256.digest())
37148 56c7cd067477 lfs: add a blob verification method to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 36932 diff changeset	237
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	238 def has(self, oid):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	239 """Returns True if the local blobstore contains the requested blob,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	240 False otherwise."""
35289 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35123 diff changeset	241 return self.cachevfs.exists(oid) or self.vfs.exists(oid)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	242
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	243 def _urlerrorreason(urlerror):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	244 '''Create a friendly message for the given URLError to be used in an
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	245 LfsRemoteError message.
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	246 '''
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	247 inst = urlerror
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	248
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	249 if isinstance(urlerror.reason, Exception):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	250 inst = urlerror.reason
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	251
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	252 if util.safehasattr(inst, 'reason'):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	253 try: # usually it is in the form (errno, strerror)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	254 reason = inst.reason.args[1]
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	255 except (AttributeError, IndexError):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	256 # it might be anything, for example a string
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	257 reason = inst.reason
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	258 if isinstance(reason, pycompat.unicode):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	259 # SSLError of Python 2.7.9 contains a unicode
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	260 reason = encoding.unitolocal(reason)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	261 return reason
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	262 elif getattr(inst, "strerror", None):
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	263 return encoding.strtolocal(inst.strerror)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	264 else:
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	265 return stringutil.forcebytestr(urlerror)
380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	266
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	267 class _gitlfsremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	268
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	269 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	270 ui = repo.ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	271 self.ui = ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	272 baseurl, authinfo = url.authinfo()
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	273 self.baseurl = baseurl.rstrip(b'/')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	274 useragent = repo.ui.config(b'experimental', b'lfs.user-agent')
35444 e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35443 diff changeset	275 if not useragent:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	276 useragent = b'git-lfs/2.3.4 (Mercurial %s)' % util.version()
35443 e7bb5fc4570c lfs: add git to the User-Agent header for blob transfers Matt Harbison <matt_harbison@yahoo.com> parents: 35437 diff changeset	277 self.urlopener = urlmod.opener(ui, authinfo, useragent)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	278 self.retry = ui.configint(b'lfs', b'retry')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	279
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	280 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	281 """Batch upload from local to remote blobstore."""
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	282 self._batch(_deduplicate(pointers), fromstore, b'upload')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	283
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	284 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	285 """Batch download from remote to local blostore."""
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	286 self._batch(_deduplicate(pointers), tostore, b'download')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	287
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	288 def _batchrequest(self, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	289 """Get metadata about objects pointed by pointers for given action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	290
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	291 Return decoded JSON object like {'objects': [{'oid': '', 'size': 1}]}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	292 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	293 """
41449 40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	294 objects = [{r'oid': pycompat.strurl(p.oid()),
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	295 r'size': p.size()} for p in pointers]
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	296 requestdata = pycompat.bytesurl(json.dumps({
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	297 r'objects': objects,
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	298 r'operation': pycompat.strurl(action),
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	299 }))
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	300 url = b'%s/objects/batch' % self.baseurl
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	301 batchreq = util.urlreq.request(pycompat.strurl(url), data=requestdata)
41449 40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	302 batchreq.add_header(r'Accept', r'application/vnd.git-lfs+json')
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	303 batchreq.add_header(r'Content-Type', r'application/vnd.git-lfs+json')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	304 try:
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	305 with contextlib.closing(self.urlopener.open(batchreq)) as rsp:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	306 rawjson = rsp.read()
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	307 except util.urlerr.httperror as ex:
40701 9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	308 hints = {
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	309 400: _(b'check that lfs serving is enabled on %s and "%s" is '
40701 9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	310 'supported') % (self.baseurl, action),
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	311 404: _(b'the "lfs.url" config may be used to override %s')
40701 9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	312 % self.baseurl,
9f78d10742af lfs: improve the hints for common errors in the Batch API Matt Harbison <matt_harbison@yahoo.com> parents: 40700 diff changeset	313 }
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	314 hint = hints.get(ex.code, _(b'api=%s, action=%s') % (url, action))
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	315 raise LfsRemoteError(
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	316 _(b'LFS HTTP error: %s') % stringutil.forcebytestr(ex),
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	317 hint=hint)
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	318 except util.urlerr.urlerror as ex:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	319 hint = (_(b'the "lfs.url" config may be used to override %s')
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	320 % self.baseurl)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	321 raise LfsRemoteError(_(b'LFS error: %s') % _urlerrorreason(ex),
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	322 hint=hint)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	323 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	324 response = json.loads(rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	325 except ValueError:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	326 raise LfsRemoteError(_(b'LFS server returns invalid JSON: %s')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	327 % rawjson.encode("utf-8"))
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	328
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	329 if self.ui.debugflag:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	330 self.ui.debug(b'Status: %d\n' % rsp.status)
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	331 # lfs-test-server and hg serve return headers in different order
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	332 headers = pycompat.bytestr(rsp.info())
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	333 self.ui.debug(b'%s\n'
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	334 % b'\n'.join(sorted(headers.splitlines())))
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	335
41449 40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	336 if r'objects' in response:
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	337 response[r'objects'] = sorted(response[r'objects'],
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	338 key=lambda p: p[r'oid'])
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	339 self.ui.debug(b'%s\n'
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	340 % pycompat.bytesurl(
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	341 json.dumps(response, indent=2,
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	342 separators=(r'', r': '),
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	343 sort_keys=True)))
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	344
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	345 return response
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	346
35666 2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	347 def _checkforservererror(self, pointers, responses, action):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	348 """Scans errors from objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	349
35694 8a23082f4d93 lfs: correct documentation typo Matt Harbison <matt_harbison@yahoo.com> parents: 35666 diff changeset	350 Raises LfsRemoteError if any objects have an error"""
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	351 for response in responses:
35666 2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	352 # The server should return 404 when objects cannot be found. Some
2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	353 # server implementation (ex. lfs-test-server) does not set "error"
2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	354 # but just removes "download" from "actions". Treat that case
2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	355 # as the same as 404 error.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	356 if b'error' not in response:
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	357 if (action == b'download'
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	358 and action not in response.get(b'actions', [])):
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	359 code = 404
35695 dd672e3d059f lfs: raise an error if the server sends an unsolicited oid Matt Harbison <matt_harbison@yahoo.com> parents: 35694 diff changeset	360 else:
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	361 continue
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	362 else:
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	363 # An error dict without a code doesn't make much sense, so
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	364 # treat as a server error.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	365 code = response.get(b'error').get(b'code', 500)
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	366
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	367 ptrmap = {p.oid(): p for p in pointers}
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	368 p = ptrmap.get(response[b'oid'], None)
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	369 if p:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	370 filename = getattr(p, 'filename', b'unknown')
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	371 errors = {
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	372 404: b'The object does not exist',
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	373 410: b'The object was removed by the owner',
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	374 422: b'Validation error',
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	375 500: b'Internal server error',
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	376 }
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	377 msg = errors.get(code, b'status code %d' % code)
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	378 raise LfsRemoteError(_(b'LFS server error for "%s": %s')
37244 67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	379 % (filename, msg))
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	380 else:
67db84842356 lfs: improve the client message when the server signals an object error Matt Harbison <matt_harbison@yahoo.com> parents: 37219 diff changeset	381 raise LfsRemoteError(
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	382 _(b'LFS server error. Unsolicited response for oid %s')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	383 % response[b'oid'])
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	384
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	385 def _extractobjects(self, response, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	386 """extract objects from response of the batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	387
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	388 response: parsed JSON object returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	389 return response['objects'] filtered by action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	390 raise if any object has an error
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	391 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	392 # Scan errors from objects - fail early
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	393 objects = response.get(b'objects', [])
35666 2c6ebd0c850e lfs: remove internal url in test Jun Wu <quark@fb.com> parents: 35614 diff changeset	394 self._checkforservererror(pointers, objects, action)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	395
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	396 # Filter objects with given action. Practically, this skips uploading
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	397 # objects which exist in the server.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	398 filteredobjects = [o for o in objects
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	399 if action in o.get(b'actions', [])]
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	400
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	401 return filteredobjects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	402
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	403 def _basictransfer(self, obj, action, localstore):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	404 """Download or upload a single object using basic transfer protocol
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	405
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	406 obj: dict, an object description returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	407 action: string, one of ['upload', 'download']
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	408 localstore: blobstore.local
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	409
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	410 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/\
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	411 basic-transfers.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	412 """
36630 4da09b46451e lfs: add some bytestring wrappers in blobstore.py Augie Fackler <augie@google.com> parents: 36485 diff changeset	413 oid = pycompat.bytestr(obj['oid'])
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	414
36630 4da09b46451e lfs: add some bytestring wrappers in blobstore.py Augie Fackler <augie@google.com> parents: 36485 diff changeset	415 href = pycompat.bytestr(obj['actions'][action].get('href'))
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	416 headers = obj['actions'][action].get('header', {}).items()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	417
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	418 request = util.urlreq.request(href)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	419 if action == b'upload':
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	420 # If uploading blobs, read data from local blobstore.
37219 b00bd974eef5 lfs: drop a duplicate blob verification method Matt Harbison <matt_harbison@yahoo.com> parents: 37149 diff changeset	421 if not localstore.verify(oid):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	422 raise error.Abort(_(b'detected corrupt lfs object: %s') % oid,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	423 hint=_(b'run hg verify'))
35530 e8f80529abeb lfs: use the local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35529 diff changeset	424 request.data = filewithprogress(localstore.open(oid), None)
41449 40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	425 request.get_method = lambda: r'PUT'
40efcf78f3df py3: raw stringify various JSON and HTTP headers in the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 41448 diff changeset	426 request.add_header(r'Content-Type', r'application/octet-stream')
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	427
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	428 for k, v in headers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	429 request.add_header(k, v)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	430
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	431 response = b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	432 try:
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	433 with contextlib.closing(self.urlopener.open(request)) as req:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	434 ui = self.ui # Shorten debug lines
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	435 if self.ui.debugflag:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	436 ui.debug(b'Status: %d\n' % req.status)
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	437 # lfs-test-server and hg serve return headers in different
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	438 # order
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	439 headers = pycompat.bytestr(req.info())
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	440 ui.debug(b'%s\n'
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	441 % b'\n'.join(sorted(headers.splitlines())))
36932 0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server Matt Harbison <matt_harbison@yahoo.com> parents: 36630 diff changeset	442
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	443 if action == b'download':
40706 fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	444 # If downloading blobs, store downloaded data to local
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	445 # blobstore
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	446 localstore.download(oid, req)
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	447 else:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	448 while True:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	449 data = req.read(1048576)
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	450 if not data:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	451 break
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	452 response += data
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed Matt Harbison <matt_harbison@yahoo.com> parents: 40703 diff changeset	453 if response:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	454 ui.debug(b'lfs %s response: %s' % (action, response))
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	455 except util.urlerr.httperror as ex:
35734 b4e1d0654736 lfs: dump the full response on httperror in debug mode Matt Harbison <matt_harbison@yahoo.com> parents: 35733 diff changeset	456 if self.ui.debugflag:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	457 self.ui.debug(b'%s: %s\n' % (oid, ex.read())) # XXX: also bytes?
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	458 raise LfsRemoteError(_(b'LFS HTTP error: %s (oid=%s, action=%s)')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	459 % (stringutil.forcebytestr(ex), oid, action))
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	460 except util.urlerr.urlerror as ex:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	461 hint = (_(b'attempted connection to %s')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	462 % pycompat.bytesurl(util.urllibcompat.getfullurl(request)))
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	463 raise LfsRemoteError(_(b'LFS error: %s') % _urlerrorreason(ex),
40702 380f5131ee7b lfs: handle URLErrors to add additional information Matt Harbison <matt_harbison@yahoo.com> parents: 40701 diff changeset	464 hint=hint)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	465
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	466 def _batch(self, pointers, localstore, action):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	467 if action not in [b'upload', b'download']:
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	468 raise error.ProgrammingError(b'invalid Git-LFS action: %s' % action)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	469
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	470 response = self._batchrequest(pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	471 objects = self._extractobjects(response, pointers, action)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	472 total = sum(x.get(b'size', 0) for x in objects)
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	473 sizes = {}
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	474 for obj in objects:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	475 sizes[obj.get(b'oid')] = obj.get(b'size', 0)
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	476 topic = {b'upload': _(b'lfs uploading'),
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	477 b'download': _(b'lfs downloading')}[action]
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	478 if len(objects) > 1:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	479 self.ui.note(_(b'lfs: need to transfer %d objects (%s)\n')
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	480 % (len(objects), util.bytecount(total)))
39416 b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	481
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	482 def transfer(chunk):
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	483 for obj in chunk:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	484 objsize = obj.get(b'size', 0)
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	485 if self.ui.verbose:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	486 if action == b'download':
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	487 msg = _(b'lfs: downloading %s (%s)\n')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	488 elif action == b'upload':
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	489 msg = _(b'lfs: uploading %s (%s)\n')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	490 self.ui.note(msg % (obj.get(b'oid'),
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	491 util.bytecount(objsize)))
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	492 retry = self.retry
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	493 while True:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	494 try:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	495 self._basictransfer(obj, action, localstore)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	496 yield 1, obj.get(b'oid')
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	497 break
35479 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	498 except socket.error as ex:
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	499 if retry > 0:
35482 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35481 diff changeset	500 self.ui.note(
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	501 _(b'lfs: failed: %r (remaining retry %d)\n')
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	502 % (stringutil.forcebytestr(ex), retry))
35437 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	503 retry -= 1
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	504 continue
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	505 raise
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35403 diff changeset	506
35732 10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	507 # Until https multiplexing gets sorted out
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	508 if self.ui.configbool(b'experimental', b'lfs.worker-enable'):
35732 10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	509 oids = worker.worker(self.ui, 0.1, transfer, (),
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	510 sorted(objects, key=lambda o: o.get(b'oid')))
35732 10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	511 else:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	512 oids = transfer(sorted(objects, key=lambda o: o.get(b'oid')))
35732 10e62d5efa73 lfs: default to not using workers for upload/download Matt Harbison <matt_harbison@yahoo.com> parents: 35695 diff changeset	513
39416 b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	514 with self.ui.makeprogress(topic, total=total) as progress:
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	515 progress.update(0)
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	516 processed = 0
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	517 blobs = 0
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	518 for _one, oid in oids:
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	519 processed += sizes[oid]
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	520 blobs += 1
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime Matt Harbison <matt_harbison@yahoo.com> parents: 38411 diff changeset	521 progress.update(processed)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	522 self.ui.note(_(b'lfs: processed: %s\n') % oid)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	523
35921 fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	524 if blobs > 0:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	525 if action == b'upload':
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	526 self.ui.status(_(b'lfs: uploaded %d files (%s)\n')
35921 fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	527 % (blobs, util.bytecount(processed)))
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	528 elif action == b'download':
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	529 self.ui.status(_(b'lfs: downloaded %d files (%s)\n')
37765 ab04972a33ef lfs: enable the final download count status message Matt Harbison <matt_harbison@yahoo.com> parents: 37692 diff changeset	530 % (blobs, util.bytecount(processed)))
35921 fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded Matt Harbison <matt_harbison@yahoo.com> parents: 35753 diff changeset	531
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	532 def __del__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	533 # copied from mercurial/httppeer.py
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	534 urlopener = getattr(self, 'urlopener', None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	535 if urlopener:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	536 for h in urlopener.handlers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	537 h.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	538 getattr(h, "close_all", lambda : None)()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	539
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	540 class _dummyremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	541 """Dummy store storing blobs to temp directory."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	542
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	543 def __init__(self, repo, url):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	544 fullpath = repo.vfs.join(b'lfs', url.path)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	545 self.vfs = lfsvfs(fullpath)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	546
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	547 def writebatch(self, pointers, fromstore):
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	548 for p in _deduplicate(pointers):
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	549 content = fromstore.read(p.oid(), verify=True)
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	550 with self.vfs(p.oid(), b'wb', atomictemp=True) as fp:
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	551 fp.write(content)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	552
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	553 def readbatch(self, pointers, tostore):
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	554 for p in _deduplicate(pointers):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	555 with self.vfs(p.oid(), b'rb') as fp:
35552 fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	556 tostore.download(p.oid(), fp)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	557
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	558 class _nullremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	559 """Null store storing blobs to /dev/null."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	560
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	561 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	562 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	563
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	564 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	565 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	566
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	567 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	568 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	569
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	570 class _promptremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	571 """Prompt user to set lfs.url when accessed."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	572
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	573 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	574 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	575
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	576 def writebatch(self, pointers, fromstore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	577 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	578
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	579 def readbatch(self, pointers, tostore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	580 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	581
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	582 def _prompt(self):
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	583 raise error.Abort(_(b'lfs.url needs to be configured'))
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	584
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	585 _storemap = {
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	586 b'https': _gitlfsremote,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	587 b'http': _gitlfsremote,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	588 b'file': _dummyremote,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	589 b'null': _nullremote,
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	590 None: _promptremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	591 }
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	592
35967 9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	593 def _deduplicate(pointers):
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	594 """Remove any duplicate oids that exist in the list"""
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	595 reduced = util.sortdict()
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	596 for p in pointers:
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	597 reduced[p.oid()] = p
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	598 return reduced.values()
9b413478f261 lfs: deduplicate oids in the transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35921 diff changeset	599
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	600 def _verify(oid, content):
40714 9fcf8084ada8 py3: use node.hex(m.digest()) instead of m.hexdigest() Pulkit Goyal <pulkit@yandex-team.ru> parents: 40706 diff changeset	601 realoid = node.hex(hashlib.sha256(content).digest())
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	602 if realoid != oid:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	603 raise LfsCorruptionError(_(b'detected corrupt lfs object: %s') % oid,
02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	604 hint=_(b'run hg verify'))
35480 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35479 diff changeset	605
37564 31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push Matt Harbison <matt_harbison@yahoo.com> parents: 37563 diff changeset	606 def remote(repo, remote=None):
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	607 """remotestore factory. return a store in _storemap depending on config
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	608
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	609 If ``lfs.url`` is specified, use that remote endpoint. Otherwise, try to
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	610 infer the endpoint, based on the remote repository using the same path
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	611 adjustments as git. As an extension, 'http' is supported as well so that
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	612 ``hg serve`` works out of the box.
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	613
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	614 https://github.com/git-lfs/git-lfs/blob/master/docs/api/server-discovery.md
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	615 """
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	616 lfsurl = repo.ui.config(b'lfs', b'url')
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	617 url = util.url(lfsurl or '')
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	618 if lfsurl is None:
37564 31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push Matt Harbison <matt_harbison@yahoo.com> parents: 37563 diff changeset	619 if remote:
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	620 path = remote
37564 31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push Matt Harbison <matt_harbison@yahoo.com> parents: 37563 diff changeset	621 elif util.safehasattr(repo, '_subtoppath'):
37563 be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	622 # The pull command sets this during the optional update phase, which
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	623 # tells exactly where the pull originated, whether 'paths.default'
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	624 # or explicit.
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	625 path = repo._subtoppath
37563 be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	626 else:
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	627 # TODO: investigate 'paths.remote:lfsurl' style path customization,
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source Matt Harbison <matt_harbison@yahoo.com> parents: 37562 diff changeset	628 # and fall back to inferring from 'paths.remote' if unspecified.
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	629 path = repo.ui.config(b'paths', b'default') or b''
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	630
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	631 defaulturl = util.url(path)
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	632
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	633 # TODO: support local paths as well.
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	634 # TODO: consider the ssh -> https transformation that git applies
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	635 if defaulturl.scheme in (b'http', b'https'):
37565 9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	636 if defaulturl.path and defaulturl.path[:-1] != b'/':
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store Matt Harbison <matt_harbison@yahoo.com> parents: 37564 diff changeset	637 defaulturl.path += b'/'
37691 d241e6632669 lfs: fix the inferred remote store path when using a --prefix Matt Harbison <matt_harbison@yahoo.com> parents: 37565 diff changeset	638 defaulturl.path = (defaulturl.path or b'') + b'.git/info/lfs'
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	639
092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	640 url = util.url(bytes(defaulturl))
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	641 repo.ui.note(_(b'lfs: assuming remote store: %s\n') % url)
37518 092eff6833a7 lfs: infer the blob store URL from paths.default Matt Harbison <matt_harbison@yahoo.com> parents: 37517 diff changeset	642
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	643 scheme = url.scheme
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	644 if scheme not in _storemap:
41448 02d0a7774882 py3: byteify the LFS blobstore module Matt Harbison <matt_harbison@yahoo.com> parents: 40714 diff changeset	645 raise error.Abort(_(b'lfs: unknown url scheme: %s') % scheme)
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	646 return _storemap[scheme](repo, url)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	647
39793 b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API) Gregory Szorc <gregory.szorc@gmail.com> parents: 39474 diff changeset	648 class LfsRemoteError(error.StorageError):
35121 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	649 pass
37692 10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	650
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	651 class LfsCorruptionError(error.Abort):
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	652 """Raised when a corrupt blob is detected, aborting an operation
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	653
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected Matt Harbison <matt_harbison@yahoo.com> parents: 37691 diff changeset	654 It exists to allow specialized handling on the server side."""

Mercurial > public > mercurial-scm > hg-stable

annotate hgext/lfs/blobstore.py @ 41449:40efcf78f3df