mercurial-scm/hg: hgext/lfs/blobstore.py annotate

annotate hgext/lfs/blobstore.py @ 35614:6d6d20658cce

lfs: drop deprecated remote store config options The last of these were removed from fb-experimental in 86884a51e9aa, and we might as well clean this up before the freeze.

author	Matt Harbison <matt_harbison@yahoo.com>
date	Fri, 12 Jan 2018 23:13:38 -0500
parents	ebf14075a5c1
children	2c6ebd0c850e

rev	line source
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	1 # blobstore.py - local and remote (speaking Git-LFS protocol) blob storages
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	2 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	3 # Copyright 2017 Facebook, Inc.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	4 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	7
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	8 from __future__ import absolute_import
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	9
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	10 import hashlib
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	11 import json
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	12 import os
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	13 import re
35475 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35473 diff changeset	14 import socket
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	15
35099 b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35098 diff changeset	16 from mercurial.i18n import _
b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35098 diff changeset	17
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	18 from mercurial import (
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	19 error,
35362 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	20 pathutil,
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	21 url as urlmod,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	22 util,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	23 vfs as vfsmod,
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	24 worker,
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	25 )
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	26
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	27 from ..largefiles import lfutil
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	28
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	29 # 64 bytes for SHA256
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	30 _lfsre = re.compile(r'\A[a-f0-9]{64}\Z')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	31
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	32 class lfsvfs(vfsmod.vfs):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	33 def join(self, path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	34 """split the path at first two characters, like: XX/XXXXX..."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	35 if not _lfsre.match(path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	36 raise error.ProgrammingError('unexpected lfs path: %s' % path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	37 return super(lfsvfs, self).join(path[0:2], path[2:])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	38
35362 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	39 def walk(self, path=None, onerror=None):
35396 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35362 diff changeset	40 """Yield (dirpath, [], oids) tuple for blobs under path
35362 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	41
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	42 Oids only exist in the root of this vfs, so dirpath is always ''.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	43 """
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	44 root = os.path.normpath(self.base)
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	45 # when dirpath == root, dirpath[prefixlen:] becomes empty
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	46 # because len(dirpath) < prefixlen.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	47 prefixlen = len(pathutil.normasprefix(root))
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	48 oids = []
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	49
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	50 for dirpath, dirs, files in os.walk(self.reljoin(self.base, path or ''),
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	51 onerror=onerror):
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	52 dirpath = dirpath[prefixlen:]
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	53
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	54 # Silently skip unexpected files and directories
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	55 if len(dirpath) == 2:
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	56 oids.extend([dirpath + f for f in files
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	57 if _lfsre.match(dirpath + f)])
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	58
35396 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35362 diff changeset	59 yield ('', [], oids)
35362 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	60
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	61 class filewithprogress(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	62 """a file-like object that supports __len__ and read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	63
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	64 Useful to provide progress information for how many bytes are read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	65 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	66
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	67 def __init__(self, fp, callback):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	68 self._fp = fp
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	69 self._callback = callback # func(readsize)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	70 fp.seek(0, os.SEEK_END)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	71 self._len = fp.tell()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	72 fp.seek(0)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	73
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	74 def __len__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	75 return self._len
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	76
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	77 def read(self, size):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	78 if self._fp is None:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	79 return b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	80 data = self._fp.read(size)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	81 if data:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	82 if self._callback:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	83 self._callback(len(data))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	84 else:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	85 self._fp.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	86 self._fp = None
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	87 return data
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	88
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	89 class local(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	90 """Local blobstore for large file contents.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	91
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	92 This blobstore is used both as a cache and as a staging area for large blobs
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	93 to be uploaded to the remote blobstore.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	94 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	95
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	96 def __init__(self, repo):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	97 fullpath = repo.svfs.join('lfs/objects')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	98 self.vfs = lfsvfs(fullpath)
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	99 usercache = lfutil._usercachedir(repo.ui, 'lfs')
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	100 self.cachevfs = lfsvfs(usercache)
35473 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35440 diff changeset	101 self.ui = repo.ui
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	102
35525 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	103 def open(self, oid):
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	104 """Open a read-only file descriptor to the named blob, in either the
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	105 usercache or the local store."""
35537 58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35526 diff changeset	106 # The usercache is the most likely place to hold the file. Commit will
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35526 diff changeset	107 # write to both it and the local store, as will anything that downloads
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35526 diff changeset	108 # the blobs. However, things like clone without an update won't
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35526 diff changeset	109 # populate the local store. For an init + push of a local clone,
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35526 diff changeset	110 # the usercache is the only place it _could_ be. If not present, the
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior Matt Harbison <matt_harbison@yahoo.com> parents: 35526 diff changeset	111 # missing file msg here will indicate the local repo, not the usercache.
35525 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	112 if self.cachevfs.exists(oid):
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	113 return self.cachevfs(oid, 'rb')
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	114
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	115 return self.vfs(oid, 'rb')
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	116
35551 fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	117 def download(self, oid, src):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	118 """Read the blob from the remote source in chunks, verify the content,
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	119 and write to this local blobstore."""
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	120 sha256 = hashlib.sha256()
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	121
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	122 with self.vfs(oid, 'wb', atomictemp=True) as fp:
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	123 for chunk in util.filechunkiter(src, size=1048576):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	124 fp.write(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	125 sha256.update(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	126
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	127 realoid = sha256.hexdigest()
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	128 if realoid != oid:
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	129 raise error.Abort(_('corrupt remote lfs object: %s') % oid)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	130
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	131 # XXX: should we verify the content of the cache, and hardlink back to
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	132 # the local store on success, but truncate, write and link on failure?
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	133 if not self.cachevfs.exists(oid):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	134 self.ui.note(_('lfs: adding %s to the usercache\n') % oid)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	135 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35537 diff changeset	136
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	137 def write(self, oid, data):
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	138 """Write blob to local blobstore.
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	139
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	140 This should only be called from the filelog during a commit or similar.
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	141 As such, there is no need to verify the data. Imports from a remote
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	142 store must use ``download()`` instead."""
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	143 with self.vfs(oid, 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	144 fp.write(data)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	145
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	146 # XXX: should we verify the content of the cache, and hardlink back to
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	147 # the local store on success, but truncate, write and link on failure?
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	148 if not self.cachevfs.exists(oid):
35553 a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	149 self.ui.note(_('lfs: adding %s to the usercache\n') % oid)
a77418095530 lfs: remove the verification option when writing to the local store Matt Harbison <matt_harbison@yahoo.com> parents: 35552 diff changeset	150 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	151
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	152 def read(self, oid, verify=True):
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	153 """Read blob from local blobstore."""
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	154 if not self.vfs.exists(oid):
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	155 blob = self._read(self.cachevfs, oid, verify)
35477 bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	156
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	157 # Even if revlog will verify the content, it needs to be verified
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	158 # now before making the hardlink to avoid propagating corrupt blobs.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	159 # Don't abort if corruption is detected, because `hg verify` will
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	160 # give more useful info about the corruption- simply don't add the
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	161 # hardlink.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	162 if verify or hashlib.sha256(blob).hexdigest() == oid:
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	163 self.ui.note(_('lfs: found %s in the usercache\n') % oid)
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	164 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
35473 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35440 diff changeset	165 else:
02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35440 diff changeset	166 self.ui.note(_('lfs: found %s in the local lfs store\n') % oid)
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	167 blob = self._read(self.vfs, oid, verify)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	168 return blob
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	169
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	170 def _read(self, vfs, oid, verify):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	171 """Read blob (after verifying) from the given store"""
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	172 blob = vfs.read(oid)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	173 if verify:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	174 _verify(oid, blob)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	175 return blob
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	176
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	177 def has(self, oid):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	178 """Returns True if the local blobstore contains the requested blob,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	179 False otherwise."""
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	180 return self.cachevfs.exists(oid) or self.vfs.exists(oid)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	181
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	182 class _gitlfsremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	183
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	184 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	185 ui = repo.ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	186 self.ui = ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	187 baseurl, authinfo = url.authinfo()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	188 self.baseurl = baseurl.rstrip('/')
35440 e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35439 diff changeset	189 useragent = repo.ui.config('experimental', 'lfs.user-agent')
e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35439 diff changeset	190 if not useragent:
e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35439 diff changeset	191 useragent = 'mercurial/%s git/2.15.1' % util.version()
35439 e7bb5fc4570c lfs: add git to the User-Agent header for blob transfers Matt Harbison <matt_harbison@yahoo.com> parents: 35433 diff changeset	192 self.urlopener = urlmod.opener(ui, authinfo, useragent)
35100 07e97998d385 lfs: register config options Matt Harbison <matt_harbison@yahoo.com> parents: 35099 diff changeset	193 self.retry = ui.configint('lfs', 'retry')
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	194
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	195 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	196 """Batch upload from local to remote blobstore."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	197 self._batch(pointers, fromstore, 'upload')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	198
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	199 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	200 """Batch download from remote to local blostore."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	201 self._batch(pointers, tostore, 'download')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	202
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	203 def _batchrequest(self, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	204 """Get metadata about objects pointed by pointers for given action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	205
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	206 Return decoded JSON object like {'objects': [{'oid': '', 'size': 1}]}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	207 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	208 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	209 objects = [{'oid': p.oid(), 'size': p.size()} for p in pointers]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	210 requestdata = json.dumps({
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	211 'objects': objects,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	212 'operation': action,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	213 })
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	214 batchreq = util.urlreq.request('%s/objects/batch' % self.baseurl,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	215 data=requestdata)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	216 batchreq.add_header('Accept', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	217 batchreq.add_header('Content-Type', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	218 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	219 rawjson = self.urlopener.open(batchreq).read()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	220 except util.urlerr.httperror as ex:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	221 raise LfsRemoteError(_('LFS HTTP error: %s (action=%s)')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	222 % (ex, action))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	223 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	224 response = json.loads(rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	225 except ValueError:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	226 raise LfsRemoteError(_('LFS server returns invalid JSON: %s')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	227 % rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	228 return response
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	229
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	230 def _checkforservererror(self, pointers, responses):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	231 """Scans errors from objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	232
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	233 Returns LfsRemoteError if any objects has an error"""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	234 for response in responses:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	235 error = response.get('error')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	236 if error:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	237 ptrmap = {p.oid(): p for p in pointers}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	238 p = ptrmap.get(response['oid'], None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	239 if error['code'] == 404 and p:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	240 filename = getattr(p, 'filename', 'unknown')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	241 raise LfsRemoteError(
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	242 _(('LFS server error. Remote object '
35568 ebf14075a5c1 lfs: improve the error message for a missing remote blob Matt Harbison <matt_harbison@yahoo.com> parents: 35553 diff changeset	243 'for "%s" not found: %r')) % (filename, response))
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	244 raise LfsRemoteError(_('LFS server error: %r') % response)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	245
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	246 def _extractobjects(self, response, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	247 """extract objects from response of the batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	248
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	249 response: parsed JSON object returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	250 return response['objects'] filtered by action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	251 raise if any object has an error
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	252 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	253 # Scan errors from objects - fail early
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	254 objects = response.get('objects', [])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	255 self._checkforservererror(pointers, objects)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	256
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	257 # Filter objects with given action. Practically, this skips uploading
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	258 # objects which exist in the server.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	259 filteredobjects = [o for o in objects if action in o.get('actions', [])]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	260 # But for downloading, we want all objects. Therefore missing objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	261 # should be considered an error.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	262 if action == 'download':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	263 if len(filteredobjects) < len(objects):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	264 missing = [o.get('oid', '?')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	265 for o in objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	266 if action not in o.get('actions', [])]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	267 raise LfsRemoteError(
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	268 _('LFS server claims required objects do not exist:\n%s')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	269 % '\n'.join(missing))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	270
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	271 return filteredobjects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	272
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	273 def _basictransfer(self, obj, action, localstore):
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	274 """Download or upload a single object using basic transfer protocol
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	275
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	276 obj: dict, an object description returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	277 action: string, one of ['upload', 'download']
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	278 localstore: blobstore.local
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	279
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	280 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/\
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	281 basic-transfers.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	282 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	283 oid = str(obj['oid'])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	284
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	285 href = str(obj['actions'][action].get('href'))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	286 headers = obj['actions'][action].get('header', {}).items()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	287
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	288 request = util.urlreq.request(href)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	289 if action == 'upload':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	290 # If uploading blobs, read data from local blobstore.
35526 e8f80529abeb lfs: use the local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35525 diff changeset	291 with localstore.open(oid) as fp:
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	292 _verifyfile(oid, fp)
35526 e8f80529abeb lfs: use the local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35525 diff changeset	293 request.data = filewithprogress(localstore.open(oid), None)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	294 request.get_method = lambda: 'PUT'
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	295
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	296 for k, v in headers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	297 request.add_header(k, v)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	298
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	299 response = b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	300 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	301 req = self.urlopener.open(request)
35552 fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	302 if action == 'download':
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	303 # If downloading blobs, store downloaded data to local blobstore
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	304 localstore.download(oid, req)
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	305 else:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	306 while True:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	307 data = req.read(1048576)
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	308 if not data:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	309 break
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	310 response += data
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	311 if response:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	312 self.ui.debug('lfs %s response: %s' % (action, response))
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	313 except util.urlerr.httperror as ex:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	314 raise LfsRemoteError(_('HTTP error: %s (oid=%s, action=%s)')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	315 % (ex, oid, action))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	316
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	317 def _batch(self, pointers, localstore, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	318 if action not in ['upload', 'download']:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	319 raise error.ProgrammingError('invalid Git-LFS action: %s' % action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	320
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	321 response = self._batchrequest(pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	322 objects = self._extractobjects(response, pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	323 total = sum(x.get('size', 0) for x in objects)
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	324 sizes = {}
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	325 for obj in objects:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	326 sizes[obj.get('oid')] = obj.get('size', 0)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	327 topic = {'upload': _('lfs uploading'),
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	328 'download': _('lfs downloading')}[action]
35478 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	329 if len(objects) > 1:
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	330 self.ui.note(_('lfs: need to transfer %d objects (%s)\n')
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	331 % (len(objects), util.bytecount(total)))
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	332 self.ui.progress(topic, 0, total=total)
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	333 def transfer(chunk):
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	334 for obj in chunk:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	335 objsize = obj.get('size', 0)
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	336 if self.ui.verbose:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	337 if action == 'download':
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	338 msg = _('lfs: downloading %s (%s)\n')
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	339 elif action == 'upload':
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	340 msg = _('lfs: uploading %s (%s)\n')
35478 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	341 self.ui.note(msg % (obj.get('oid'),
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	342 util.bytecount(objsize)))
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	343 retry = self.retry
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	344 while True:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	345 try:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	346 self._basictransfer(obj, action, localstore)
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	347 yield 1, obj.get('oid')
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	348 break
35475 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35473 diff changeset	349 except socket.error as ex:
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	350 if retry > 0:
35478 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	351 self.ui.note(
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	352 _('lfs: failed: %r (remaining retry %d)\n')
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	353 % (ex, retry))
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	354 retry -= 1
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	355 continue
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	356 raise
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	357
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	358 oids = worker.worker(self.ui, 0.1, transfer, (),
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	359 sorted(objects, key=lambda o: o.get('oid')))
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	360 processed = 0
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	361 for _one, oid in oids:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	362 processed += sizes[oid]
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	363 self.ui.progress(topic, processed, total=total)
35478 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	364 self.ui.note(_('lfs: processed: %s\n') % oid)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	365 self.ui.progress(topic, pos=None, total=total)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	366
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	367 def __del__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	368 # copied from mercurial/httppeer.py
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	369 urlopener = getattr(self, 'urlopener', None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	370 if urlopener:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	371 for h in urlopener.handlers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	372 h.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	373 getattr(h, "close_all", lambda : None)()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	374
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	375 class _dummyremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	376 """Dummy store storing blobs to temp directory."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	377
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	378 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	379 fullpath = repo.vfs.join('lfs', url.path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	380 self.vfs = lfsvfs(fullpath)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	381
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	382 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	383 for p in pointers:
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	384 content = fromstore.read(p.oid(), verify=True)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	385 with self.vfs(p.oid(), 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	386 fp.write(content)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	387
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	388 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	389 for p in pointers:
35552 fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	390 with self.vfs(p.oid(), 'rb') as fp:
fd610befc37f lfs: use the localstore download method to transfer from remote stores Matt Harbison <matt_harbison@yahoo.com> parents: 35551 diff changeset	391 tostore.download(p.oid(), fp)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	392
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	393 class _nullremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	394 """Null store storing blobs to /dev/null."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	395
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	396 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	397 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	398
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	399 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	400 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	401
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	402 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	403 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	404
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	405 class _promptremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	406 """Prompt user to set lfs.url when accessed."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	407
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	408 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	409 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	410
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	411 def writebatch(self, pointers, fromstore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	412 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	413
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	414 def readbatch(self, pointers, tostore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	415 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	416
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	417 def _prompt(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	418 raise error.Abort(_('lfs.url needs to be configured'))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	419
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	420 _storemap = {
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	421 'https': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	422 'http': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	423 'file': _dummyremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	424 'null': _nullremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	425 None: _promptremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	426 }
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	427
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	428 def _verify(oid, content):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	429 realoid = hashlib.sha256(content).hexdigest()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	430 if realoid != oid:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	431 raise error.Abort(_('detected corrupt lfs object: %s') % oid,
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	432 hint=_('run hg verify'))
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	433
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	434 def _verifyfile(oid, fp):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	435 sha256 = hashlib.sha256()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	436 while True:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	437 data = fp.read(1024 * 1024)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	438 if not data:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	439 break
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	440 sha256.update(data)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	441 realoid = sha256.hexdigest()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	442 if realoid != oid:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	443 raise error.Abort(_('detected corrupt lfs object: %s') % oid,
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	444 hint=_('run hg verify'))
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	445
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	446 def remote(repo):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	447 """remotestore factory. return a store in _storemap depending on config"""
35614 6d6d20658cce lfs: drop deprecated remote store config options Matt Harbison <matt_harbison@yahoo.com> parents: 35568 diff changeset	448 url = util.url(repo.ui.config('lfs', 'url') or '')
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	449 scheme = url.scheme
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	450 if scheme not in _storemap:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	451 raise error.Abort(_('lfs: unknown url scheme: %s') % scheme)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	452 return _storemap[scheme](repo, url)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	453
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	454 class LfsRemoteError(error.RevlogError):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	455 pass

Mercurial > public > mercurial-scm > hg

annotate hgext/lfs/blobstore.py @ 35614:6d6d20658cce