mercurial-scm/hg: hgext/lfs/blobstore.py annotate

annotate hgext/lfs/blobstore.py @ 35526:e8f80529abeb

lfs: use the local store method for opening a blob I noticed that when I cloned without updating and then turned around and pushed that clone to an lfs server, it was only trying to find the blob in the local store. Writes to the dummyremote (file based store) use local.read(), which looks at both the usercache and local store.

author	Matt Harbison <matt_harbison@yahoo.com>
date	Tue, 02 Jan 2018 21:46:57 -0500
parents	83903433c2eb
children	58fda95a0202

rev	line source
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	1 # blobstore.py - local and remote (speaking Git-LFS protocol) blob storages
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	2 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	3 # Copyright 2017 Facebook, Inc.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	4 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	7
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	8 from __future__ import absolute_import
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	9
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	10 import hashlib
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	11 import json
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	12 import os
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	13 import re
35475 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35473 diff changeset	14 import socket
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	15
35099 b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35098 diff changeset	16 from mercurial.i18n import _
b8e5fb8d2389 lfs: quiesce check-module-import warnings Matt Harbison <matt_harbison@yahoo.com> parents: 35098 diff changeset	17
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	18 from mercurial import (
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	19 error,
35362 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	20 pathutil,
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	21 url as urlmod,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	22 util,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	23 vfs as vfsmod,
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	24 worker,
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	25 )
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	26
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	27 from ..largefiles import lfutil
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	28
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	29 # 64 bytes for SHA256
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	30 _lfsre = re.compile(r'\A[a-f0-9]{64}\Z')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	31
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	32 class lfsvfs(vfsmod.vfs):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	33 def join(self, path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	34 """split the path at first two characters, like: XX/XXXXX..."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	35 if not _lfsre.match(path):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	36 raise error.ProgrammingError('unexpected lfs path: %s' % path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	37 return super(lfsvfs, self).join(path[0:2], path[2:])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	38
35362 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	39 def walk(self, path=None, onerror=None):
35396 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35362 diff changeset	40 """Yield (dirpath, [], oids) tuple for blobs under path
35362 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	41
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	42 Oids only exist in the root of this vfs, so dirpath is always ''.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	43 """
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	44 root = os.path.normpath(self.base)
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	45 # when dirpath == root, dirpath[prefixlen:] becomes empty
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	46 # because len(dirpath) < prefixlen.
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	47 prefixlen = len(pathutil.normasprefix(root))
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	48 oids = []
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	49
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	50 for dirpath, dirs, files in os.walk(self.reljoin(self.base, path or ''),
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	51 onerror=onerror):
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	52 dirpath = dirpath[prefixlen:]
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	53
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	54 # Silently skip unexpected files and directories
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	55 if len(dirpath) == 2:
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	56 oids.extend([dirpath + f for f in files
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	57 if _lfsre.match(dirpath + f)])
79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	58
35396 c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk() Matt Harbison <matt_harbison@yahoo.com> parents: 35362 diff changeset	59 yield ('', [], oids)
35362 79968f91ad0c lfs: override walk() in lfsvfs Matt Harbison <matt_harbison@yahoo.com> parents: 35280 diff changeset	60
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	61 class filewithprogress(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	62 """a file-like object that supports __len__ and read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	63
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	64 Useful to provide progress information for how many bytes are read.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	65 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	66
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	67 def __init__(self, fp, callback):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	68 self._fp = fp
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	69 self._callback = callback # func(readsize)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	70 fp.seek(0, os.SEEK_END)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	71 self._len = fp.tell()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	72 fp.seek(0)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	73
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	74 def __len__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	75 return self._len
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	76
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	77 def read(self, size):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	78 if self._fp is None:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	79 return b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	80 data = self._fp.read(size)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	81 if data:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	82 if self._callback:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	83 self._callback(len(data))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	84 else:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	85 self._fp.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	86 self._fp = None
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	87 return data
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	88
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	89 class local(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	90 """Local blobstore for large file contents.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	91
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	92 This blobstore is used both as a cache and as a staging area for large blobs
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	93 to be uploaded to the remote blobstore.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	94 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	95
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	96 def __init__(self, repo):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	97 fullpath = repo.svfs.join('lfs/objects')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	98 self.vfs = lfsvfs(fullpath)
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	99 usercache = lfutil._usercachedir(repo.ui, 'lfs')
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	100 self.cachevfs = lfsvfs(usercache)
35473 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35440 diff changeset	101 self.ui = repo.ui
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	102
35525 83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	103 def open(self, oid):
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	104 """Open a read-only file descriptor to the named blob, in either the
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	105 usercache or the local store."""
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	106 if self.cachevfs.exists(oid):
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	107 return self.cachevfs(oid, 'rb')
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	108
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	109 return self.vfs(oid, 'rb')
83903433c2eb lfs: add a local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35478 diff changeset	110
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	111 def write(self, oid, data, verify=True):
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	112 """Write blob to local blobstore."""
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	113 if verify:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	114 _verify(oid, data)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	115
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	116 with self.vfs(oid, 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	117 fp.write(data)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	118
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	119 # XXX: should we verify the content of the cache, and hardlink back to
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	120 # the local store on success, but truncate, write and link on failure?
8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	121 if not self.cachevfs.exists(oid):
35477 bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	122 if verify or hashlib.sha256(data).hexdigest() == oid:
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	123 self.ui.note(_('lfs: adding %s to the usercache\n') % oid)
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	124 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	125
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	126 def read(self, oid, verify=True):
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	127 """Read blob from local blobstore."""
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	128 if not self.vfs.exists(oid):
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	129 blob = self._read(self.cachevfs, oid, verify)
35477 bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	130
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	131 # Even if revlog will verify the content, it needs to be verified
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	132 # now before making the hardlink to avoid propagating corrupt blobs.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	133 # Don't abort if corruption is detected, because `hg verify` will
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	134 # give more useful info about the corruption- simply don't add the
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	135 # hardlink.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	136 if verify or hashlib.sha256(blob).hexdigest() == oid:
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	137 self.ui.note(_('lfs: found %s in the usercache\n') % oid)
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies Matt Harbison <matt_harbison@yahoo.com> parents: 35476 diff changeset	138 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
35473 02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35440 diff changeset	139 else:
02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob Matt Harbison <matt_harbison@yahoo.com> parents: 35440 diff changeset	140 self.ui.note(_('lfs: found %s in the local lfs store\n') % oid)
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	141 blob = self._read(self.vfs, oid, verify)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	142 return blob
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	143
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	144 def _read(self, vfs, oid, verify):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	145 """Read blob (after verifying) from the given store"""
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	146 blob = vfs.read(oid)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	147 if verify:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	148 _verify(oid, blob)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	149 return blob
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	150
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	151 def has(self, oid):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	152 """Returns True if the local blobstore contains the requested blob,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	153 False otherwise."""
35280 8e72f9152c4d lfs: introduce a user level cache for lfs files Matt Harbison <matt_harbison@yahoo.com> parents: 35100 diff changeset	154 return self.cachevfs.exists(oid) or self.vfs.exists(oid)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	155
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	156 class _gitlfsremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	157
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	158 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	159 ui = repo.ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	160 self.ui = ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	161 baseurl, authinfo = url.authinfo()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	162 self.baseurl = baseurl.rstrip('/')
35440 e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35439 diff changeset	163 useragent = repo.ui.config('experimental', 'lfs.user-agent')
e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35439 diff changeset	164 if not useragent:
e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer Matt Harbison <matt_harbison@yahoo.com> parents: 35439 diff changeset	165 useragent = 'mercurial/%s git/2.15.1' % util.version()
35439 e7bb5fc4570c lfs: add git to the User-Agent header for blob transfers Matt Harbison <matt_harbison@yahoo.com> parents: 35433 diff changeset	166 self.urlopener = urlmod.opener(ui, authinfo, useragent)
35100 07e97998d385 lfs: register config options Matt Harbison <matt_harbison@yahoo.com> parents: 35099 diff changeset	167 self.retry = ui.configint('lfs', 'retry')
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	168
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	169 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	170 """Batch upload from local to remote blobstore."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	171 self._batch(pointers, fromstore, 'upload')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	172
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	173 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	174 """Batch download from remote to local blostore."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	175 self._batch(pointers, tostore, 'download')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	176
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	177 def _batchrequest(self, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	178 """Get metadata about objects pointed by pointers for given action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	179
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	180 Return decoded JSON object like {'objects': [{'oid': '', 'size': 1}]}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	181 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	182 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	183 objects = [{'oid': p.oid(), 'size': p.size()} for p in pointers]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	184 requestdata = json.dumps({
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	185 'objects': objects,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	186 'operation': action,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	187 })
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	188 batchreq = util.urlreq.request('%s/objects/batch' % self.baseurl,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	189 data=requestdata)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	190 batchreq.add_header('Accept', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	191 batchreq.add_header('Content-Type', 'application/vnd.git-lfs+json')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	192 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	193 rawjson = self.urlopener.open(batchreq).read()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	194 except util.urlerr.httperror as ex:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	195 raise LfsRemoteError(_('LFS HTTP error: %s (action=%s)')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	196 % (ex, action))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	197 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	198 response = json.loads(rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	199 except ValueError:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	200 raise LfsRemoteError(_('LFS server returns invalid JSON: %s')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	201 % rawjson)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	202 return response
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	203
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	204 def _checkforservererror(self, pointers, responses):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	205 """Scans errors from objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	206
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	207 Returns LfsRemoteError if any objects has an error"""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	208 for response in responses:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	209 error = response.get('error')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	210 if error:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	211 ptrmap = {p.oid(): p for p in pointers}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	212 p = ptrmap.get(response['oid'], None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	213 if error['code'] == 404 and p:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	214 filename = getattr(p, 'filename', 'unknown')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	215 raise LfsRemoteError(
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	216 _(('LFS server error. Remote object '
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	217 'for file %s not found: %r')) % (filename, response))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	218 raise LfsRemoteError(_('LFS server error: %r') % response)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	219
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	220 def _extractobjects(self, response, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	221 """extract objects from response of the batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	222
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	223 response: parsed JSON object returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	224 return response['objects'] filtered by action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	225 raise if any object has an error
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	226 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	227 # Scan errors from objects - fail early
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	228 objects = response.get('objects', [])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	229 self._checkforservererror(pointers, objects)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	230
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	231 # Filter objects with given action. Practically, this skips uploading
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	232 # objects which exist in the server.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	233 filteredobjects = [o for o in objects if action in o.get('actions', [])]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	234 # But for downloading, we want all objects. Therefore missing objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	235 # should be considered an error.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	236 if action == 'download':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	237 if len(filteredobjects) < len(objects):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	238 missing = [o.get('oid', '?')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	239 for o in objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	240 if action not in o.get('actions', [])]
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	241 raise LfsRemoteError(
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	242 _('LFS server claims required objects do not exist:\n%s')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	243 % '\n'.join(missing))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	244
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	245 return filteredobjects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	246
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	247 def _basictransfer(self, obj, action, localstore):
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	248 """Download or upload a single object using basic transfer protocol
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	249
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	250 obj: dict, an object description returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	251 action: string, one of ['upload', 'download']
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	252 localstore: blobstore.local
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	253
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	254 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/\
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	255 basic-transfers.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	256 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	257 oid = str(obj['oid'])
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	258
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	259 href = str(obj['actions'][action].get('href'))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	260 headers = obj['actions'][action].get('header', {}).items()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	261
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	262 request = util.urlreq.request(href)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	263 if action == 'upload':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	264 # If uploading blobs, read data from local blobstore.
35526 e8f80529abeb lfs: use the local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35525 diff changeset	265 with localstore.open(oid) as fp:
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	266 _verifyfile(oid, fp)
35526 e8f80529abeb lfs: use the local store method for opening a blob Matt Harbison <matt_harbison@yahoo.com> parents: 35525 diff changeset	267 request.data = filewithprogress(localstore.open(oid), None)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	268 request.get_method = lambda: 'PUT'
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	269
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	270 for k, v in headers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	271 request.add_header(k, v)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	272
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	273 response = b''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	274 try:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	275 req = self.urlopener.open(request)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	276 while True:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	277 data = req.read(1048576)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	278 if not data:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	279 break
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	280 response += data
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	281 except util.urlerr.httperror as ex:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	282 raise LfsRemoteError(_('HTTP error: %s (oid=%s, action=%s)')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	283 % (ex, oid, action))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	284
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	285 if action == 'download':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	286 # If downloading blobs, store downloaded data to local blobstore
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	287 localstore.write(oid, response, verify=True)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	288
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	289 def _batch(self, pointers, localstore, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	290 if action not in ['upload', 'download']:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	291 raise error.ProgrammingError('invalid Git-LFS action: %s' % action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	292
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	293 response = self._batchrequest(pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	294 objects = self._extractobjects(response, pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	295 total = sum(x.get('size', 0) for x in objects)
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	296 sizes = {}
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	297 for obj in objects:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	298 sizes[obj.get('oid')] = obj.get('size', 0)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	299 topic = {'upload': _('lfs uploading'),
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	300 'download': _('lfs downloading')}[action]
35478 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	301 if len(objects) > 1:
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	302 self.ui.note(_('lfs: need to transfer %d objects (%s)\n')
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	303 % (len(objects), util.bytecount(total)))
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	304 self.ui.progress(topic, 0, total=total)
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	305 def transfer(chunk):
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	306 for obj in chunk:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	307 objsize = obj.get('size', 0)
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	308 if self.ui.verbose:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	309 if action == 'download':
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	310 msg = _('lfs: downloading %s (%s)\n')
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	311 elif action == 'upload':
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	312 msg = _('lfs: uploading %s (%s)\n')
35478 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	313 self.ui.note(msg % (obj.get('oid'),
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	314 util.bytecount(objsize)))
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	315 retry = self.retry
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	316 while True:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	317 try:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	318 self._basictransfer(obj, action, localstore)
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	319 yield 1, obj.get('oid')
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	320 break
35475 b0c01a5ee35c lfs: narrow the exceptions that trigger a transfer retry Matt Harbison <matt_harbison@yahoo.com> parents: 35473 diff changeset	321 except socket.error as ex:
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	322 if retry > 0:
35478 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	323 self.ui.note(
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	324 _('lfs: failed: %r (remaining retry %d)\n')
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	325 % (ex, retry))
35433 f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	326 retry -= 1
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	327 continue
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	328 raise
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	329
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	330 oids = worker.worker(self.ui, 0.1, transfer, (),
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	331 sorted(objects, key=lambda o: o.get('oid')))
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	332 processed = 0
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	333 for _one, oid in oids:
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	334 processed += sizes[oid]
f98fac24b757 lfs: using workers in lfs prefetch Wojciech Lis <wlis@fb.com> parents: 35396 diff changeset	335 self.ui.progress(topic, processed, total=total)
35478 5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Matt Harbison <matt_harbison@yahoo.com> parents: 35477 diff changeset	336 self.ui.note(_('lfs: processed: %s\n') % oid)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	337 self.ui.progress(topic, pos=None, total=total)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	338
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	339 def __del__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	340 # copied from mercurial/httppeer.py
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	341 urlopener = getattr(self, 'urlopener', None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	342 if urlopener:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	343 for h in urlopener.handlers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	344 h.close()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	345 getattr(h, "close_all", lambda : None)()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	346
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	347 class _dummyremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	348 """Dummy store storing blobs to temp directory."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	349
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	350 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	351 fullpath = repo.vfs.join('lfs', url.path)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	352 self.vfs = lfsvfs(fullpath)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	353
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	354 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	355 for p in pointers:
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	356 content = fromstore.read(p.oid(), verify=True)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	357 with self.vfs(p.oid(), 'wb', atomictemp=True) as fp:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	358 fp.write(content)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	359
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	360 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	361 for p in pointers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	362 content = self.vfs.read(p.oid())
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	363 tostore.write(p.oid(), content, verify=True)
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	364
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	365 class _nullremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	366 """Null store storing blobs to /dev/null."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	367
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	368 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	369 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	370
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	371 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	372 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	373
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	374 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	375 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	376
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	377 class _promptremote(object):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	378 """Prompt user to set lfs.url when accessed."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	379
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	380 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	381 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	382
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	383 def writebatch(self, pointers, fromstore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	384 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	385
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	386 def readbatch(self, pointers, tostore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	387 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	388
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	389 def _prompt(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	390 raise error.Abort(_('lfs.url needs to be configured'))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	391
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	392 _storemap = {
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	393 'https': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	394 'http': _gitlfsremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	395 'file': _dummyremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	396 'null': _nullremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	397 None: _promptremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	398 }
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	399
35476 417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	400 def _verify(oid, content):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	401 realoid = hashlib.sha256(content).hexdigest()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	402 if realoid != oid:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	403 raise error.Abort(_('detected corrupt lfs object: %s') % oid,
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	404 hint=_('run hg verify'))
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	405
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	406 def _verifyfile(oid, fp):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	407 sha256 = hashlib.sha256()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	408 while True:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	409 data = fp.read(1024 * 1024)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	410 if not data:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	411 break
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	412 sha256.update(data)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	413 realoid = sha256.hexdigest()
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	414 if realoid != oid:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	415 raise error.Abort(_('detected corrupt lfs object: %s') % oid,
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	416 hint=_('run hg verify'))
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store Matt Harbison <matt_harbison@yahoo.com> parents: 35475 diff changeset	417
35098 66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	418 def remote(repo):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	419 """remotestore factory. return a store in _storemap depending on config"""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	420 defaulturl = ''
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	421
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	422 # convert deprecated configs to the new url. TODO: remove this if other
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	423 # places are migrated to the new url config.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	424 # deprecated config: lfs.remotestore
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	425 deprecatedstore = repo.ui.config('lfs', 'remotestore')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	426 if deprecatedstore == 'dummy':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	427 # deprecated config: lfs.remotepath
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	428 defaulturl = 'file://' + repo.ui.config('lfs', 'remotepath')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	429 elif deprecatedstore == 'git-lfs':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	430 # deprecated config: lfs.remoteurl
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	431 defaulturl = repo.ui.config('lfs', 'remoteurl')
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	432 elif deprecatedstore == 'null':
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	433 defaulturl = 'null://'
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	434
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	435 url = util.url(repo.ui.config('lfs', 'url', defaulturl))
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	436 scheme = url.scheme
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	437 if scheme not in _storemap:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	438 raise error.Abort(_('lfs: unknown url scheme: %s') % scheme)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	439 return _storemap[scheme](repo, url)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	440
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	441 class LfsRemoteError(error.RevlogError):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension Matt Harbison <matt_harbison@yahoo.com> parents: diff changeset	442 pass

Mercurial > public > mercurial-scm > hg

annotate hgext/lfs/blobstore.py @ 35526:e8f80529abeb