annotate hgext/lfs/blobstore.py @ 52640:24ee91ba9aa8

pyupgrade: drop usage of py3 aliases for `OSError` These were different classes in py2, but now a handful of error classes are just an alias of `OSError`, like `IOError`, `EnvironmentError`, `WindowsError`, etc. This is the result of running a hacked version of `pyupgrade` 3.19.1[1] $ hg files -0 'relglob:**.py' | xargs -0 \ pyupgrade --py38-plus --keep-percent-format --keep-mock --keep-runtime-typing The hack is because it doesn't have command line switches to disable most changes, so it makes tons of unrelated changes all at once. The hack is to 1) patch `pyupgrade._main._fix_tokens()` to immediately return its content arg 2) change `pyupgrade._data.register_decorator()` to only register the function if it's from the fixer we're interested in: if func.__module__ in ( "pyupgrade._plugins.exceptions", ): FUNCS[tp].append(func) return func [1] https://github.com/asottile/pyupgrade
author Matt Harbison <matt_harbison@yahoo.com>
date Sun, 05 Jan 2025 21:03:17 -0500
parents d62887764687
children 5cc8deb96b48
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
1 # blobstore.py - local and remote (speaking Git-LFS protocol) blob storages
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
2 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
3 # Copyright 2017 Facebook, Inc.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
4 #
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
7
51859
f4733654f144 typing: add `from __future__ import annotations` to most files
Matt Harbison <matt_harbison@yahoo.com>
parents: 50926
diff changeset
8 from __future__ import annotations
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
9
40665
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
10 import contextlib
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
11 import errno
35476
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
12 import hashlib
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
13 import json
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
14 import os
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
15 import re
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
16
51877
d62887764687 lfs: fix various signature mismatches for vfs subclasses
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
17 from typing import (
d62887764687 lfs: fix various signature mismatches for vfs subclasses
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
18 Optional,
d62887764687 lfs: fix various signature mismatches for vfs subclasses
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
19 )
d62887764687 lfs: fix various signature mismatches for vfs subclasses
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
20
35099
b8e5fb8d2389 lfs: quiesce check-module-import warnings
Matt Harbison <matt_harbison@yahoo.com>
parents: 35098
diff changeset
21 from mercurial.i18n import _
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
22 from mercurial.node import hex
35099
b8e5fb8d2389 lfs: quiesce check-module-import warnings
Matt Harbison <matt_harbison@yahoo.com>
parents: 35098
diff changeset
23
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
24 from mercurial import (
40661
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
25 encoding,
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
26 error,
44273
43eea17ae7b3 lfs: fix the stall and corruption issue when concurrently uploading blobs
Matt Harbison <matt_harbison@yahoo.com>
parents: 44272
diff changeset
27 httpconnection as httpconnectionmod,
35362
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
28 pathutil,
36601
4da09b46451e lfs: add some bytestring wrappers in blobstore.py
Augie Fackler <augie@google.com>
parents: 36455
diff changeset
29 pycompat,
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
30 url as urlmod,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
31 util,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
32 vfs as vfsmod,
35433
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
33 worker,
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
34 )
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
35
46907
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46113
diff changeset
36 from mercurial.utils import (
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46113
diff changeset
37 stringutil,
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46113
diff changeset
38 urlutil,
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46113
diff changeset
39 )
40661
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
40
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
41 from ..largefiles import lfutil
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
42
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
43 # 64 bytes for SHA256
36455
9e3cb58c7ab3 py3: make sure regexes are bytes
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35927
diff changeset
44 _lfsre = re.compile(br'\A[a-f0-9]{64}\Z')
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
45
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
46
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
47 class lfsvfs(vfsmod.vfs):
51877
d62887764687 lfs: fix various signature mismatches for vfs subclasses
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
48 def join(self, path: Optional[bytes], *insidef: bytes) -> bytes:
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
49 """split the path at first two characters, like: XX/XXXXX..."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
50 if not _lfsre.match(path):
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
51 raise error.ProgrammingError(b'unexpected lfs path: %s' % path)
51877
d62887764687 lfs: fix various signature mismatches for vfs subclasses
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
52 return super(lfsvfs, self).join(path[0:2], path[2:], *insidef)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
53
35362
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
54 def walk(self, path=None, onerror=None):
35396
c8edeb03ca94 lfs: correct the directory list value returned by lfsvfs.walk()
Matt Harbison <matt_harbison@yahoo.com>
parents: 35362
diff changeset
55 """Yield (dirpath, [], oids) tuple for blobs under path
35362
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
56
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
57 Oids only exist in the root of this vfs, so dirpath is always ''.
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
58 """
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
59 root = os.path.normpath(self.base)
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
60 # when dirpath == root, dirpath[prefixlen:] becomes empty
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
61 # because len(dirpath) < prefixlen.
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
62 prefixlen = len(pathutil.normasprefix(root))
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
63 oids = []
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
64
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
65 for dirpath, dirs, files in os.walk(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
66 self.reljoin(self.base, path or b''), onerror=onerror
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
67 ):
35362
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
68 dirpath = dirpath[prefixlen:]
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
69
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
70 # Silently skip unexpected files and directories
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
71 if len(dirpath) == 2:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
72 oids.extend(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
73 [dirpath + f for f in files if _lfsre.match(dirpath + f)]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
74 )
35362
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
75
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
76 yield (b'', [], oids)
35362
79968f91ad0c lfs: override walk() in lfsvfs
Matt Harbison <matt_harbison@yahoo.com>
parents: 35280
diff changeset
77
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
78
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
79 class nullvfs(lfsvfs):
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
80 def __init__(self):
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
81 pass
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
82
51877
d62887764687 lfs: fix various signature mismatches for vfs subclasses
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
83 def exists(self, path: Optional[bytes] = None) -> bool:
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
84 return False
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
85
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
86 def read(self, oid):
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
87 # store.read() calls into here if the blob doesn't exist in its
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
88 # self.vfs. Raise the same error as a normal vfs when asked to read a
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
89 # file that doesn't exist. The only difference is the full file path
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
90 # isn't available in the error.
52640
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51877
diff changeset
91 raise OSError(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
92 errno.ENOENT,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
93 pycompat.sysstr(b'%s: No such file or directory' % oid),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
94 )
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
95
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
96 def walk(self, path=None, onerror=None):
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
97 return (b'', [], [])
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
98
51877
d62887764687 lfs: fix various signature mismatches for vfs subclasses
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
99 def write(self, *args, **kwargs) -> int:
d62887764687 lfs: fix various signature mismatches for vfs subclasses
Matt Harbison <matt_harbison@yahoo.com>
parents: 51859
diff changeset
100 return 0
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
101
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
102
44273
43eea17ae7b3 lfs: fix the stall and corruption issue when concurrently uploading blobs
Matt Harbison <matt_harbison@yahoo.com>
parents: 44272
diff changeset
103 class lfsuploadfile(httpconnectionmod.httpsendfile):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44294
diff changeset
104 """a file-like object that supports keepalive."""
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
105
44273
43eea17ae7b3 lfs: fix the stall and corruption issue when concurrently uploading blobs
Matt Harbison <matt_harbison@yahoo.com>
parents: 44272
diff changeset
106 def __init__(self, ui, filename):
43eea17ae7b3 lfs: fix the stall and corruption issue when concurrently uploading blobs
Matt Harbison <matt_harbison@yahoo.com>
parents: 44272
diff changeset
107 super(lfsuploadfile, self).__init__(ui, filename, b'rb')
43eea17ae7b3 lfs: fix the stall and corruption issue when concurrently uploading blobs
Matt Harbison <matt_harbison@yahoo.com>
parents: 44272
diff changeset
108 self.read = self._data.read
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
109
44273
43eea17ae7b3 lfs: fix the stall and corruption issue when concurrently uploading blobs
Matt Harbison <matt_harbison@yahoo.com>
parents: 44272
diff changeset
110 def _makeprogress(self):
43eea17ae7b3 lfs: fix the stall and corruption issue when concurrently uploading blobs
Matt Harbison <matt_harbison@yahoo.com>
parents: 44272
diff changeset
111 return None # progress is handled by the worker client
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
112
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
113
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48934
diff changeset
114 class local:
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
115 """Local blobstore for large file contents.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
116
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
117 This blobstore is used both as a cache and as a staging area for large blobs
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
118 to be uploaded to the remote blobstore.
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
119 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
120
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
121 def __init__(self, repo):
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
122 fullpath = repo.svfs.join(b'lfs/objects')
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
123 self.vfs = lfsvfs(fullpath)
37562
e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url
Matt Harbison <matt_harbison@yahoo.com>
parents: 37518
diff changeset
124
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
125 if repo.ui.configbool(b'experimental', b'lfs.disableusercache'):
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
126 self.cachevfs = nullvfs()
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
127 else:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
128 usercache = lfutil._usercachedir(repo.ui, b'lfs')
37562
e5cd8d1a094d lfs: special case the null:// usercache instead of treating it as a url
Matt Harbison <matt_harbison@yahoo.com>
parents: 37518
diff changeset
129 self.cachevfs = lfsvfs(usercache)
35473
02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35440
diff changeset
130 self.ui = repo.ui
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
131
35525
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35478
diff changeset
132 def open(self, oid):
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35478
diff changeset
133 """Open a read-only file descriptor to the named blob, in either the
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35478
diff changeset
134 usercache or the local store."""
44294
234001d22ba6 lfs: use str for the open() mode when opening a blob for py3
Matt Harbison <matt_harbison@yahoo.com>
parents: 44273
diff changeset
135 return open(self.path(oid), 'rb')
44272
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
136
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
137 def path(self, oid):
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
138 """Build the path for the given blob ``oid``.
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
139
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
140 If the blob exists locally, the path may point to either the usercache
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
141 or the local store. If it doesn't, it will point to the local store.
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
142 This is meant for situations where existing code that isn't LFS aware
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
143 needs to open a blob. Generally, prefer the ``open`` method on this
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
144 class.
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
145 """
35537
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35526
diff changeset
146 # The usercache is the most likely place to hold the file. Commit will
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35526
diff changeset
147 # write to both it and the local store, as will anything that downloads
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35526
diff changeset
148 # the blobs. However, things like clone without an update won't
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35526
diff changeset
149 # populate the local store. For an init + push of a local clone,
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35526
diff changeset
150 # the usercache is the only place it _could_ be. If not present, the
58fda95a0202 lfs: add a comment to describe subtle local blobstore open() behavior
Matt Harbison <matt_harbison@yahoo.com>
parents: 35526
diff changeset
151 # missing file msg here will indicate the local repo, not the usercache.
35525
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35478
diff changeset
152 if self.cachevfs.exists(oid):
44272
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
153 return self.cachevfs.join(oid)
35525
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35478
diff changeset
154
44272
06de4a673f48 lfs: add a method to the local blobstore to convert OIDs to file paths
Matt Harbison <matt_harbison@yahoo.com>
parents: 44141
diff changeset
155 return self.vfs.join(oid)
35525
83903433c2eb lfs: add a local store method for opening a blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35478
diff changeset
156
44085
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
157 def download(self, oid, src, content_length):
35551
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
158 """Read the blob from the remote source in chunks, verify the content,
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
159 and write to this local blobstore."""
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
160 sha256 = hashlib.sha256()
44085
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
161 size = 0
35551
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
162
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
163 with self.vfs(oid, b'wb', atomictemp=True) as fp:
35551
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
164 for chunk in util.filechunkiter(src, size=1048576):
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
165 fp.write(chunk)
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
166 sha256.update(chunk)
44085
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
167 size += len(chunk)
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
168
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
169 # If the server advertised a length longer than what we actually
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
170 # received, then we should expect that the server crashed while
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
171 # producing the response (but the server has no way of telling us
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
172 # that), and we really don't need to try to write the response to
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
173 # the localstore, because it's not going to match the expected.
49532
250d9c8aaf10 lfs: improve an exception message for blob corruption detected on transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
174 # The server also uses this method to store data uploaded by the
250d9c8aaf10 lfs: improve an exception message for blob corruption detected on transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
175 # client, so if this happens on the server side, it's possible
250d9c8aaf10 lfs: improve an exception message for blob corruption detected on transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
176 # that the client crashed or an antivirus interfered with the
250d9c8aaf10 lfs: improve an exception message for blob corruption detected on transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
177 # upload.
44085
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
178 if content_length is not None and int(content_length) != size:
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
179 msg = (
49526
192949b68159 lfs: fix interpolation of int and %s in an exception case
Matt Harbison <matt_harbison@yahoo.com>
parents: 48946
diff changeset
180 b"Response length (%d) does not match Content-Length "
49532
250d9c8aaf10 lfs: improve an exception message for blob corruption detected on transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
181 b"header (%d) for %s"
44085
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
182 )
49532
250d9c8aaf10 lfs: improve an exception message for blob corruption detected on transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
183 raise LfsRemoteError(_(msg) % (size, int(content_length), oid))
35551
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
184
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
185 realoid = hex(sha256.digest())
35551
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
186 if realoid != oid:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
187 raise LfsCorruptionError(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
188 _(b'corrupt remote lfs object: %s') % oid
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
189 )
35551
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
190
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
191 self._linktousercache(oid)
35551
fa9dd53eb23e lfs: introduce a localstore method for downloading from remote stores
Matt Harbison <matt_harbison@yahoo.com>
parents: 35537
diff changeset
192
35553
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
193 def write(self, oid, data):
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
194 """Write blob to local blobstore.
35476
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
195
35553
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
196 This should only be called from the filelog during a commit or similar.
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
197 As such, there is no need to verify the data. Imports from a remote
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
198 store must use ``download()`` instead."""
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
199 with self.vfs(oid, b'wb', atomictemp=True) as fp:
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
200 fp.write(data)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
201
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
202 self._linktousercache(oid)
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
203
39457
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
204 def linkfromusercache(self, oid):
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
205 """Link blobs found in the user cache into this store.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
206
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
207 The server module needs to do this when it lets the client know not to
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
208 upload the blob, to ensure it is always available in this store.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
209 Normally this is done implicitly when the client reads or writes the
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
210 blob, but that doesn't happen when the server tells the client that it
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
211 already has the blob.
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
212 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
213 if not isinstance(self.cachevfs, nullvfs) and not self.vfs.exists(oid):
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
214 self.ui.note(_(b'lfs: found %s in the usercache\n') % oid)
39457
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
215 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
a913d2892e17 lfs: ensure the blob is linked to the remote store on skipped uploads
Matt Harbison <matt_harbison@yahoo.com>
parents: 39389
diff changeset
216
37517
491edf2435a0 lfs: add the ability to disable the usercache
Matt Harbison <matt_harbison@yahoo.com>
parents: 37243
diff changeset
217 def _linktousercache(self, oid):
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
218 # XXX: should we verify the content of the cache, and hardlink back to
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
219 # the local store on success, but truncate, write and link on failure?
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
220 if not self.cachevfs.exists(oid) and not isinstance(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
221 self.cachevfs, nullvfs
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
222 ):
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
223 self.ui.note(_(b'lfs: adding %s to the usercache\n') % oid)
35553
a77418095530 lfs: remove the verification option when writing to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35552
diff changeset
224 lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
225
35476
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
226 def read(self, oid, verify=True):
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
227 """Read blob from local blobstore."""
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
228 if not self.vfs.exists(oid):
35476
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
229 blob = self._read(self.cachevfs, oid, verify)
35477
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35476
diff changeset
230
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35476
diff changeset
231 # Even if revlog will verify the content, it needs to be verified
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35476
diff changeset
232 # now before making the hardlink to avoid propagating corrupt blobs.
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35476
diff changeset
233 # Don't abort if corruption is detected, because `hg verify` will
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35476
diff changeset
234 # give more useful info about the corruption- simply don't add the
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35476
diff changeset
235 # hardlink.
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
236 if verify or hex(hashlib.sha256(blob).digest()) == oid:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
237 self.ui.note(_(b'lfs: found %s in the usercache\n') % oid)
35477
bb6a80fc969a lfs: only hardlink between the usercache and local store if the blob verifies
Matt Harbison <matt_harbison@yahoo.com>
parents: 35476
diff changeset
238 lfutil.link(self.cachevfs.join(oid), self.vfs.join(oid))
35473
02f54a1ec9eb lfs: add note messages indicating what store holds the lfs blob
Matt Harbison <matt_harbison@yahoo.com>
parents: 35440
diff changeset
239 else:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
240 self.ui.note(_(b'lfs: found %s in the local lfs store\n') % oid)
35476
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
241 blob = self._read(self.vfs, oid, verify)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
242 return blob
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
243
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
244 def _read(self, vfs, oid, verify):
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
245 """Read blob (after verifying) from the given store"""
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
246 blob = vfs.read(oid)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
247 if verify:
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
248 _verify(oid, blob)
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
249 return blob
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
250
37145
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
251 def verify(self, oid):
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
252 """Indicate whether or not the hash of the underlying file matches its
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
253 name."""
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
254 sha256 = hashlib.sha256()
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
255
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
256 with self.open(oid) as fp:
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
257 for chunk in util.filechunkiter(fp, size=1048576):
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
258 sha256.update(chunk)
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
259
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
260 return oid == hex(sha256.digest())
37145
56c7cd067477 lfs: add a blob verification method to the local store
Matt Harbison <matt_harbison@yahoo.com>
parents: 36926
diff changeset
261
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
262 def has(self, oid):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
263 """Returns True if the local blobstore contains the requested blob,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
264 False otherwise."""
35280
8e72f9152c4d lfs: introduce a user level cache for lfs files
Matt Harbison <matt_harbison@yahoo.com>
parents: 35100
diff changeset
265 return self.cachevfs.exists(oid) or self.vfs.exists(oid)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
266
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
267
40661
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
268 def _urlerrorreason(urlerror):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44294
diff changeset
269 """Create a friendly message for the given URLError to be used in an
40661
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
270 LfsRemoteError message.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44294
diff changeset
271 """
40661
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
272 inst = urlerror
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
273
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
274 if isinstance(urlerror.reason, Exception):
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
275 inst = urlerror.reason
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
276
50925
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50898
diff changeset
277 if hasattr(inst, 'reason'):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
278 try: # usually it is in the form (errno, strerror)
40661
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
279 reason = inst.reason.args[1]
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
280 except (AttributeError, IndexError):
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
281 # it might be anything, for example a string
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
282 reason = inst.reason
48934
06de08b36c82 py3: use str instead of pycompat.unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
283 if isinstance(reason, str):
40661
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
284 # SSLError of Python 2.7.9 contains a unicode
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
285 reason = encoding.unitolocal(reason)
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
286 return reason
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
287 elif getattr(inst, "strerror", None):
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
288 return encoding.strtolocal(inst.strerror)
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
289 else:
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
290 return stringutil.forcebytestr(urlerror)
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
291
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
292
41607
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
293 class lfsauthhandler(util.urlreq.basehandler):
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
294 handler_order = 480 # Before HTTPDigestAuthHandler (== 490)
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
295
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
296 def http_error_401(self, req, fp, code, msg, headers):
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
297 """Enforces that any authentication performed is HTTP Basic
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
298 Authentication. No authentication is also acceptable.
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
299 """
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43380
diff changeset
300 authreq = headers.get('www-authenticate', None)
41607
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
301 if authreq:
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
302 scheme = authreq.split()[0]
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
303
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43380
diff changeset
304 if scheme.lower() != 'basic':
41607
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
305 msg = _(b'the server must support Basic Authentication')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
306 raise util.urlerr.httperror(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
307 req.get_full_url(),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
308 code,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
309 encoding.strfromlocal(msg),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
310 headers,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
311 fp,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
312 )
41607
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
313 return None
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
314
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
315
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48934
diff changeset
316 class _gitlfsremote:
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
317 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
318 ui = repo.ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
319 self.ui = ui
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
320 baseurl, authinfo = url.authinfo()
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
321 self.baseurl = baseurl.rstrip(b'/')
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
322 useragent = repo.ui.config(b'experimental', b'lfs.user-agent')
35440
e333d27514b0 lfs: add an experimental config to override User-Agent for the blob transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35439
diff changeset
323 if not useragent:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
324 useragent = b'git-lfs/2.3.4 (Mercurial %s)' % util.version()
35439
e7bb5fc4570c lfs: add git to the User-Agent header for blob transfers
Matt Harbison <matt_harbison@yahoo.com>
parents: 35433
diff changeset
325 self.urlopener = urlmod.opener(ui, authinfo, useragent)
41607
698667eb7523 lfs: disable all authentication except Basic for HTTP(S) connections
Matt Harbison <matt_harbison@yahoo.com>
parents: 41440
diff changeset
326 self.urlopener.add_handler(lfsauthhandler())
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
327 self.retry = ui.configint(b'lfs', b'retry')
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
328
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
329 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
330 """Batch upload from local to remote blobstore."""
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
331 self._batch(_deduplicate(pointers), fromstore, b'upload')
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
332
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
333 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
334 """Batch download from remote to local blostore."""
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
335 self._batch(_deduplicate(pointers), tostore, b'download')
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
336
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
337 def _batchrequest(self, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
338 """Get metadata about objects pointed by pointers for given action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
339
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
340 Return decoded JSON object like {'objects': [{'oid': '', 'size': 1}]}
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
341 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
342 """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
343 objects = [
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43380
diff changeset
344 {'oid': pycompat.strurl(p.oid()), 'size': p.size()}
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
345 for p in pointers
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
346 ]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
347 requestdata = pycompat.bytesurl(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
348 json.dumps(
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44294
diff changeset
349 {
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44294
diff changeset
350 'objects': objects,
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44294
diff changeset
351 'operation': pycompat.strurl(action),
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44294
diff changeset
352 }
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
353 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
354 )
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
355 url = b'%s/objects/batch' % self.baseurl
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
356 batchreq = util.urlreq.request(pycompat.strurl(url), data=requestdata)
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43380
diff changeset
357 batchreq.add_header('Accept', 'application/vnd.git-lfs+json')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43380
diff changeset
358 batchreq.add_header('Content-Type', 'application/vnd.git-lfs+json')
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
359 try:
40665
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
360 with contextlib.closing(self.urlopener.open(batchreq)) as rsp:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
361 rawjson = rsp.read()
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
362 except util.urlerr.httperror as ex:
40660
9f78d10742af lfs: improve the hints for common errors in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 40659
diff changeset
363 hints = {
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
364 400: _(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
365 b'check that lfs serving is enabled on %s and "%s" is '
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
366 b'supported'
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
367 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
368 % (self.baseurl, action),
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
369 404: _(b'the "lfs.url" config may be used to override %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
370 % self.baseurl,
40660
9f78d10742af lfs: improve the hints for common errors in the Batch API
Matt Harbison <matt_harbison@yahoo.com>
parents: 40659
diff changeset
371 }
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
372 hint = hints.get(ex.code, _(b'api=%s, action=%s') % (url, action))
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
373 raise LfsRemoteError(
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
374 _(b'LFS HTTP error: %s') % stringutil.forcebytestr(ex),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
375 hint=hint,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
376 )
40661
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
377 except util.urlerr.urlerror as ex:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
378 hint = (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
379 _(b'the "lfs.url" config may be used to override %s')
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
380 % self.baseurl
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
381 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
382 raise LfsRemoteError(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
383 _(b'LFS error: %s') % _urlerrorreason(ex), hint=hint
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
384 )
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
385 try:
43380
579672b347d2 py3: define and use json.loads polyfill
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43091
diff changeset
386 response = pycompat.json_loads(rawjson)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
387 except ValueError:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
388 raise LfsRemoteError(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
389 _(b'LFS server returns invalid JSON: %s')
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
390 % rawjson.encode("utf-8")
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
391 )
36926
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36601
diff changeset
392
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36601
diff changeset
393 if self.ui.debugflag:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
394 self.ui.debug(b'Status: %d\n' % rsp.status)
36926
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36601
diff changeset
395 # lfs-test-server and hg serve return headers in different order
41431
0b636d1720a0 lfs: strip the response headers from the Batch API before printing
Matt Harbison <matt_harbison@yahoo.com>
parents: 41429
diff changeset
396 headers = pycompat.bytestr(rsp.info()).strip()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
397 self.ui.debug(b'%s\n' % b'\n'.join(sorted(headers.splitlines())))
36926
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36601
diff changeset
398
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43380
diff changeset
399 if 'objects' in response:
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43380
diff changeset
400 response['objects'] = sorted(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43380
diff changeset
401 response['objects'], key=lambda p: p['oid']
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
402 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
403 self.ui.debug(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
404 b'%s\n'
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
405 % pycompat.bytesurl(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
406 json.dumps(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
407 response,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
408 indent=2,
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43380
diff changeset
409 separators=('', ': '),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
410 sort_keys=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
411 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
412 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
413 )
36926
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36601
diff changeset
414
41429
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
Matt Harbison <matt_harbison@yahoo.com>
parents: 41427
diff changeset
415 def encodestr(x):
48934
06de08b36c82 py3: use str instead of pycompat.unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
416 if isinstance(x, str):
43091
127cc1f72e70 py3: stop normalizing .encode()/.decode() arguments to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43089
diff changeset
417 return x.encode('utf-8')
41429
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
Matt Harbison <matt_harbison@yahoo.com>
parents: 41427
diff changeset
418 return x
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
Matt Harbison <matt_harbison@yahoo.com>
parents: 41427
diff changeset
419
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
Matt Harbison <matt_harbison@yahoo.com>
parents: 41427
diff changeset
420 return pycompat.rapply(encodestr, response)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
421
35666
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
422 def _checkforservererror(self, pointers, responses, action):
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
423 """Scans errors from objects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
424
35694
8a23082f4d93 lfs: correct documentation typo
Matt Harbison <matt_harbison@yahoo.com>
parents: 35666
diff changeset
425 Raises LfsRemoteError if any objects have an error"""
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
426 for response in responses:
35666
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
427 # The server should return 404 when objects cannot be found. Some
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
428 # server implementation (ex. lfs-test-server) does not set "error"
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
429 # but just removes "download" from "actions". Treat that case
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
430 # as the same as 404 error.
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
431 if b'error' not in response:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
432 if action == b'download' and action not in response.get(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
433 b'actions', []
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
434 ):
37242
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
435 code = 404
35695
dd672e3d059f lfs: raise an error if the server sends an unsolicited oid
Matt Harbison <matt_harbison@yahoo.com>
parents: 35694
diff changeset
436 else:
37242
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
437 continue
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
438 else:
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
439 # An error dict without a code doesn't make much sense, so
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
440 # treat as a server error.
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
441 code = response.get(b'error').get(b'code', 500)
37242
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
442
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
443 ptrmap = {p.oid(): p for p in pointers}
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
444 p = ptrmap.get(response[b'oid'], None)
37242
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
445 if p:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
446 filename = getattr(p, 'filename', b'unknown')
37242
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
447 errors = {
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
448 404: b'The object does not exist',
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
449 410: b'The object was removed by the owner',
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
450 422: b'Validation error',
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
451 500: b'Internal server error',
37242
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
452 }
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
453 msg = errors.get(code, b'status code %d' % code)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
454 raise LfsRemoteError(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
455 _(b'LFS server error for "%s": %s') % (filename, msg)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
456 )
37242
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
457 else:
67db84842356 lfs: improve the client message when the server signals an object error
Matt Harbison <matt_harbison@yahoo.com>
parents: 37217
diff changeset
458 raise LfsRemoteError(
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
459 _(b'LFS server error. Unsolicited response for oid %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
460 % response[b'oid']
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
461 )
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
462
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
463 def _extractobjects(self, response, pointers, action):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
464 """extract objects from response of the batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
465
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
466 response: parsed JSON object returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
467 return response['objects'] filtered by action
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
468 raise if any object has an error
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
469 """
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
470 # Scan errors from objects - fail early
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
471 objects = response.get(b'objects', [])
35666
2c6ebd0c850e lfs: remove internal url in test
Jun Wu <quark@fb.com>
parents: 35614
diff changeset
472 self._checkforservererror(pointers, objects, action)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
473
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
474 # Filter objects with given action. Practically, this skips uploading
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
475 # objects which exist in the server.
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
476 filteredobjects = [
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
477 o for o in objects if action in o.get(b'actions', [])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
478 ]
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
479
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
480 return filteredobjects
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
481
35433
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
482 def _basictransfer(self, obj, action, localstore):
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
483 """Download or upload a single object using basic transfer protocol
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
484
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
485 obj: dict, an object description returned by batch API
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
486 action: string, one of ['upload', 'download']
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
487 localstore: blobstore.local
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
488
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
489 See https://github.com/git-lfs/git-lfs/blob/master/docs/api/\
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
490 basic-transfers.md
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
491 """
41429
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
Matt Harbison <matt_harbison@yahoo.com>
parents: 41427
diff changeset
492 oid = obj[b'oid']
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
Matt Harbison <matt_harbison@yahoo.com>
parents: 41427
diff changeset
493 href = obj[b'actions'][action].get(b'href')
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
Matt Harbison <matt_harbison@yahoo.com>
parents: 41427
diff changeset
494 headers = obj[b'actions'][action].get(b'header', {}).items()
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
495
41429
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
Matt Harbison <matt_harbison@yahoo.com>
parents: 41427
diff changeset
496 request = util.urlreq.request(pycompat.strurl(href))
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
497 if action == b'upload':
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
498 # If uploading blobs, read data from local blobstore.
37217
b00bd974eef5 lfs: drop a duplicate blob verification method
Matt Harbison <matt_harbison@yahoo.com>
parents: 37146
diff changeset
499 if not localstore.verify(oid):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
500 raise error.Abort(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
501 _(b'detected corrupt lfs object: %s') % oid,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
502 hint=_(b'run hg verify'),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
503 )
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
504
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
505 for k, v in headers:
41429
7df10ea7a5b8 py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
Matt Harbison <matt_harbison@yahoo.com>
parents: 41427
diff changeset
506 request.add_header(pycompat.strurl(k), pycompat.strurl(v))
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
507
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
508 try:
44141
46c8f15fb2b4 lfs: move the initialization of the upload request into the try block
Matt Harbison <matt_harbison@yahoo.com>
parents: 44140
diff changeset
509 if action == b'upload':
44273
43eea17ae7b3 lfs: fix the stall and corruption issue when concurrently uploading blobs
Matt Harbison <matt_harbison@yahoo.com>
parents: 44272
diff changeset
510 request.data = lfsuploadfile(self.ui, localstore.path(oid))
44141
46c8f15fb2b4 lfs: move the initialization of the upload request into the try block
Matt Harbison <matt_harbison@yahoo.com>
parents: 44140
diff changeset
511 request.get_method = lambda: 'PUT'
46c8f15fb2b4 lfs: move the initialization of the upload request into the try block
Matt Harbison <matt_harbison@yahoo.com>
parents: 44140
diff changeset
512 request.add_header('Content-Type', 'application/octet-stream')
44273
43eea17ae7b3 lfs: fix the stall and corruption issue when concurrently uploading blobs
Matt Harbison <matt_harbison@yahoo.com>
parents: 44272
diff changeset
513 request.add_header('Content-Length', request.data.length)
44141
46c8f15fb2b4 lfs: move the initialization of the upload request into the try block
Matt Harbison <matt_harbison@yahoo.com>
parents: 44140
diff changeset
514
44084
84f2becbd106 lfs: rename a variable to clarify its use
Matt Harbison <matt_harbison@yahoo.com>
parents: 44077
diff changeset
515 with contextlib.closing(self.urlopener.open(request)) as res:
44085
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
516 contentlength = res.info().get(b"content-length")
40665
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
517 ui = self.ui # Shorten debug lines
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
518 if self.ui.debugflag:
44084
84f2becbd106 lfs: rename a variable to clarify its use
Matt Harbison <matt_harbison@yahoo.com>
parents: 44077
diff changeset
519 ui.debug(b'Status: %d\n' % res.status)
40665
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
520 # lfs-test-server and hg serve return headers in different
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
521 # order
44084
84f2becbd106 lfs: rename a variable to clarify its use
Matt Harbison <matt_harbison@yahoo.com>
parents: 44077
diff changeset
522 headers = pycompat.bytestr(res.info()).strip()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
523 ui.debug(b'%s\n' % b'\n'.join(sorted(headers.splitlines())))
36926
0dcf50dc90b6 lfs: debug print HTTP headers and JSON payload received from the server
Matt Harbison <matt_harbison@yahoo.com>
parents: 36601
diff changeset
524
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
525 if action == b'download':
40665
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
526 # If downloading blobs, store downloaded data to local
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
527 # blobstore
44085
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
528 localstore.download(oid, res, contentlength)
40665
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
529 else:
44086
ffac09da7a19 lfs: avoid quadratic performance in processing server responses
Matt Harbison <matt_harbison@yahoo.com>
parents: 44085
diff changeset
530 blocks = []
40665
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
531 while True:
44084
84f2becbd106 lfs: rename a variable to clarify its use
Matt Harbison <matt_harbison@yahoo.com>
parents: 44077
diff changeset
532 data = res.read(1048576)
40665
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
533 if not data:
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
534 break
44086
ffac09da7a19 lfs: avoid quadratic performance in processing server responses
Matt Harbison <matt_harbison@yahoo.com>
parents: 44085
diff changeset
535 blocks.append(data)
ffac09da7a19 lfs: avoid quadratic performance in processing server responses
Matt Harbison <matt_harbison@yahoo.com>
parents: 44085
diff changeset
536
ffac09da7a19 lfs: avoid quadratic performance in processing server responses
Matt Harbison <matt_harbison@yahoo.com>
parents: 44085
diff changeset
537 response = b"".join(blocks)
40665
fb379b78b93e lfs: ensure that the return of urlopener.open() is closed
Matt Harbison <matt_harbison@yahoo.com>
parents: 40662
diff changeset
538 if response:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
539 ui.debug(b'lfs %s response: %s' % (action, response))
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
540 except util.urlerr.httperror as ex:
35734
b4e1d0654736 lfs: dump the full response on httperror in debug mode
Matt Harbison <matt_harbison@yahoo.com>
parents: 35733
diff changeset
541 if self.ui.debugflag:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
542 self.ui.debug(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
543 b'%s: %s\n' % (oid, ex.read())
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
544 ) # XXX: also bytes?
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
545 raise LfsRemoteError(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
546 _(b'LFS HTTP error: %s (oid=%s, action=%s)')
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
547 % (stringutil.forcebytestr(ex), oid, action)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
548 )
40661
380f5131ee7b lfs: handle URLErrors to add additional information
Matt Harbison <matt_harbison@yahoo.com>
parents: 40660
diff changeset
549 except util.urlerr.urlerror as ex:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
550 hint = _(b'attempted connection to %s') % pycompat.bytesurl(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
551 util.urllibcompat.getfullurl(request)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
552 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
553 raise LfsRemoteError(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
554 _(b'LFS error: %s') % _urlerrorreason(ex), hint=hint
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
555 )
44139
2ad4e8aefcf4 lfs: explicitly close the file handle for the blob being uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 44138
diff changeset
556 finally:
2ad4e8aefcf4 lfs: explicitly close the file handle for the blob being uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 44138
diff changeset
557 if request.data:
2ad4e8aefcf4 lfs: explicitly close the file handle for the blob being uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 44138
diff changeset
558 request.data.close()
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
559
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
560 def _batch(self, pointers, localstore, action):
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
561 if action not in [b'upload', b'download']:
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
562 raise error.ProgrammingError(b'invalid Git-LFS action: %s' % action)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
563
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
564 response = self._batchrequest(pointers, action)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
565 objects = self._extractobjects(response, pointers, action)
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
566 total = sum(x.get(b'size', 0) for x in objects)
35433
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
567 sizes = {}
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
568 for obj in objects:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
569 sizes[obj.get(b'oid')] = obj.get(b'size', 0)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
570 topic = {
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
571 b'upload': _(b'lfs uploading'),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
572 b'download': _(b'lfs downloading'),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
573 }[action]
35478
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35477
diff changeset
574 if len(objects) > 1:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
575 self.ui.note(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
576 _(b'lfs: need to transfer %d objects (%s)\n')
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
577 % (len(objects), util.bytecount(total))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
578 )
39389
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38405
diff changeset
579
35433
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
580 def transfer(chunk):
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
581 for obj in chunk:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
582 objsize = obj.get(b'size', 0)
35433
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
583 if self.ui.verbose:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
584 if action == b'download':
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
585 msg = _(b'lfs: downloading %s (%s)\n')
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
586 elif action == b'upload':
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
587 msg = _(b'lfs: uploading %s (%s)\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
588 self.ui.note(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
589 msg % (obj.get(b'oid'), util.bytecount(objsize))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
590 )
35433
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
591 retry = self.retry
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
592 while True:
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
593 try:
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
594 self._basictransfer(obj, action, localstore)
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
595 yield 1, obj.get(b'oid')
35433
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
596 break
52640
24ee91ba9aa8 pyupgrade: drop usage of py3 aliases for `OSError`
Matt Harbison <matt_harbison@yahoo.com>
parents: 51877
diff changeset
597 except OSError as ex:
35433
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
598 if retry > 0:
35478
5a73a0446afd lfs: use ui.note() and ui.debug() instead of ui.write() and their flags
Matt Harbison <matt_harbison@yahoo.com>
parents: 35477
diff changeset
599 self.ui.note(
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
600 _(b'lfs: failed: %r (remaining retry %d)\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
601 % (stringutil.forcebytestr(ex), retry)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
602 )
35433
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
603 retry -= 1
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
604 continue
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
605 raise
f98fac24b757 lfs: using workers in lfs prefetch
Wojciech Lis <wlis@fb.com>
parents: 35396
diff changeset
606
49545
3556f0392808 lfs: avoid closing connections when the worker doesn't fork
Matt Harbison <matt_harbison@yahoo.com>
parents: 49544
diff changeset
607 # Until https multiplexing gets sorted out. It's not clear if
3556f0392808 lfs: avoid closing connections when the worker doesn't fork
Matt Harbison <matt_harbison@yahoo.com>
parents: 49544
diff changeset
608 # ConnectionManager.set_ready() is externally synchronized for thread
3556f0392808 lfs: avoid closing connections when the worker doesn't fork
Matt Harbison <matt_harbison@yahoo.com>
parents: 49544
diff changeset
609 # safety with Windows workers.
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
610 if self.ui.configbool(b'experimental', b'lfs.worker-enable'):
49544
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
611 # The POSIX workers are forks of this process, so before spinning
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
612 # them up, close all pooled connections. Otherwise, there's no way
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
613 # to coordinate between them about who is using what, and the
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
614 # transfers will get corrupted.
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
615 #
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
616 # TODO: add a function to keepalive.ConnectionManager to mark all
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
617 # ready connections as in use, and roll that back after the fork?
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
618 # That would allow the existing pool of connections in this process
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
619 # to be preserved.
49545
3556f0392808 lfs: avoid closing connections when the worker doesn't fork
Matt Harbison <matt_harbison@yahoo.com>
parents: 49544
diff changeset
620 def prefork():
49544
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
621 for h in self.urlopener.handlers:
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
622 getattr(h, "close_all", lambda: None)()
abf471862b8e lfs: fix blob corruption when tranferring with workers on posix
Matt Harbison <matt_harbison@yahoo.com>
parents: 49526
diff changeset
623
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
624 oids = worker.worker(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
625 self.ui,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
626 0.1,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
627 transfer,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
628 (),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
629 sorted(objects, key=lambda o: o.get(b'oid')),
49545
3556f0392808 lfs: avoid closing connections when the worker doesn't fork
Matt Harbison <matt_harbison@yahoo.com>
parents: 49544
diff changeset
630 prefork=prefork,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
631 )
35732
10e62d5efa73 lfs: default to not using workers for upload/download
Matt Harbison <matt_harbison@yahoo.com>
parents: 35695
diff changeset
632 else:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
633 oids = transfer(sorted(objects, key=lambda o: o.get(b'oid')))
35732
10e62d5efa73 lfs: default to not using workers for upload/download
Matt Harbison <matt_harbison@yahoo.com>
parents: 35695
diff changeset
634
44077
05881d002cb2 lfs: add "bytes" as the unit to the upload/download progress bar
Matt Harbison <matt_harbison@yahoo.com>
parents: 43506
diff changeset
635 with self.ui.makeprogress(
05881d002cb2 lfs: add "bytes" as the unit to the upload/download progress bar
Matt Harbison <matt_harbison@yahoo.com>
parents: 43506
diff changeset
636 topic, unit=_(b"bytes"), total=total
05881d002cb2 lfs: add "bytes" as the unit to the upload/download progress bar
Matt Harbison <matt_harbison@yahoo.com>
parents: 43506
diff changeset
637 ) as progress:
39389
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38405
diff changeset
638 progress.update(0)
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38405
diff changeset
639 processed = 0
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38405
diff changeset
640 blobs = 0
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38405
diff changeset
641 for _one, oid in oids:
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38405
diff changeset
642 processed += sizes[oid]
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38405
diff changeset
643 blobs += 1
b26350d9d7b5 lfs: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38405
diff changeset
644 progress.update(processed)
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
645 self.ui.note(_(b'lfs: processed: %s\n') % oid)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
646
35881
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
647 if blobs > 0:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
648 if action == b'upload':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
649 self.ui.status(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
650 _(b'lfs: uploaded %d files (%s)\n')
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
651 % (blobs, util.bytecount(processed))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
652 )
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
653 elif action == b'download':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
654 self.ui.status(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
655 _(b'lfs: downloaded %d files (%s)\n')
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
656 % (blobs, util.bytecount(processed))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
657 )
35881
fa993c3c8462 lfs: emit a status message to indicate how many blobs were uploaded
Matt Harbison <matt_harbison@yahoo.com>
parents: 35753
diff changeset
658
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
659 def __del__(self):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
660 # copied from mercurial/httppeer.py
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
661 urlopener = getattr(self, 'urlopener', None)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
662 if urlopener:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
663 for h in urlopener.handlers:
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
664 h.close()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
665 getattr(h, "close_all", lambda: None)()
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
666
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
667
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48934
diff changeset
668 class _dummyremote:
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
669 """Dummy store storing blobs to temp directory."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
670
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
671 def __init__(self, repo, url):
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
672 fullpath = repo.vfs.join(b'lfs', url.path)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
673 self.vfs = lfsvfs(fullpath)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
674
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
675 def writebatch(self, pointers, fromstore):
35927
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35881
diff changeset
676 for p in _deduplicate(pointers):
35476
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
677 content = fromstore.read(p.oid(), verify=True)
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
678 with self.vfs(p.oid(), b'wb', atomictemp=True) as fp:
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
679 fp.write(content)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
680
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
681 def readbatch(self, pointers, tostore):
35927
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35881
diff changeset
682 for p in _deduplicate(pointers):
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
683 with self.vfs(p.oid(), b'rb') as fp:
44085
0ee0a3f6a990 lfs: check content length after downloading content
Matt Harbison <matt_harbison@yahoo.com>
parents: 44084
diff changeset
684 tostore.download(p.oid(), fp, None)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
685
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
686
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48934
diff changeset
687 class _nullremote:
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
688 """Null store storing blobs to /dev/null."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
689
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
690 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
691 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
692
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
693 def writebatch(self, pointers, fromstore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
694 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
695
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
696 def readbatch(self, pointers, tostore):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
697 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
698
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
699
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48934
diff changeset
700 class _promptremote:
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
701 """Prompt user to set lfs.url when accessed."""
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
702
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
703 def __init__(self, repo, url):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
704 pass
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
705
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
706 def writebatch(self, pointers, fromstore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
707 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
708
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
709 def readbatch(self, pointers, tostore, ui=None):
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
710 self._prompt()
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
711
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
712 def _prompt(self):
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
713 raise error.Abort(_(b'lfs.url needs to be configured'))
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
714
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
715
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
716 _storemap = {
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
717 b'https': _gitlfsremote,
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
718 b'http': _gitlfsremote,
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
719 b'file': _dummyremote,
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
720 b'null': _nullremote,
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
721 None: _promptremote,
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
722 }
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
723
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
724
35927
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35881
diff changeset
725 def _deduplicate(pointers):
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35881
diff changeset
726 """Remove any duplicate oids that exist in the list"""
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35881
diff changeset
727 reduced = util.sortdict()
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35881
diff changeset
728 for p in pointers:
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35881
diff changeset
729 reduced[p.oid()] = p
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35881
diff changeset
730 return reduced.values()
9b413478f261 lfs: deduplicate oids in the transfer
Matt Harbison <matt_harbison@yahoo.com>
parents: 35881
diff changeset
731
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
732
35476
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
733 def _verify(oid, content):
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
734 realoid = hex(hashlib.sha256(content).digest())
35476
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
735 if realoid != oid:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
736 raise LfsCorruptionError(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
737 _(b'detected corrupt lfs object: %s') % oid,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
738 hint=_(b'run hg verify'),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
739 )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
740
35476
417e8e040102 lfs: verify lfs object content when transferring to and from the remote store
Matt Harbison <matt_harbison@yahoo.com>
parents: 35475
diff changeset
741
37564
31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push
Matt Harbison <matt_harbison@yahoo.com>
parents: 37563
diff changeset
742 def remote(repo, remote=None):
37518
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
743 """remotestore factory. return a store in _storemap depending on config
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
744
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
745 If ``lfs.url`` is specified, use that remote endpoint. Otherwise, try to
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
746 infer the endpoint, based on the remote repository using the same path
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
747 adjustments as git. As an extension, 'http' is supported as well so that
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
748 ``hg serve`` works out of the box.
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
749
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
750 https://github.com/git-lfs/git-lfs/blob/master/docs/api/server-discovery.md
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
751 """
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
752 lfsurl = repo.ui.config(b'lfs', b'url')
46907
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46113
diff changeset
753 url = urlutil.url(lfsurl or b'')
37565
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store
Matt Harbison <matt_harbison@yahoo.com>
parents: 37564
diff changeset
754 if lfsurl is None:
37564
31a4ea773369 lfs: infer the blob store URL from an explicit push dest or default-push
Matt Harbison <matt_harbison@yahoo.com>
parents: 37563
diff changeset
755 if remote:
37565
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store
Matt Harbison <matt_harbison@yahoo.com>
parents: 37564
diff changeset
756 path = remote
50925
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50898
diff changeset
757 elif hasattr(repo, '_subtoppath'):
37563
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source
Matt Harbison <matt_harbison@yahoo.com>
parents: 37562
diff changeset
758 # The pull command sets this during the optional update phase, which
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source
Matt Harbison <matt_harbison@yahoo.com>
parents: 37562
diff changeset
759 # tells exactly where the pull originated, whether 'paths.default'
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source
Matt Harbison <matt_harbison@yahoo.com>
parents: 37562
diff changeset
760 # or explicit.
37565
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store
Matt Harbison <matt_harbison@yahoo.com>
parents: 37564
diff changeset
761 path = repo._subtoppath
37563
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source
Matt Harbison <matt_harbison@yahoo.com>
parents: 37562
diff changeset
762 else:
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source
Matt Harbison <matt_harbison@yahoo.com>
parents: 37562
diff changeset
763 # TODO: investigate 'paths.remote:lfsurl' style path customization,
be1cc65bdb1c lfs: infer the blob store URL from an explicit pull source
Matt Harbison <matt_harbison@yahoo.com>
parents: 37562
diff changeset
764 # and fall back to inferring from 'paths.remote' if unspecified.
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
765 path = repo.ui.config(b'paths', b'default') or b''
37565
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store
Matt Harbison <matt_harbison@yahoo.com>
parents: 37564
diff changeset
766
46907
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46113
diff changeset
767 defaulturl = urlutil.url(path)
37518
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
768
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
769 # TODO: support local paths as well.
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
770 # TODO: consider the ssh -> https transformation that git applies
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
771 if defaulturl.scheme in (b'http', b'https'):
37565
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store
Matt Harbison <matt_harbison@yahoo.com>
parents: 37564
diff changeset
772 if defaulturl.path and defaulturl.path[:-1] != b'/':
9c7a25ef5b49 lfs: handle paths that don't end with '/' when inferring the blob store
Matt Harbison <matt_harbison@yahoo.com>
parents: 37564
diff changeset
773 defaulturl.path += b'/'
37691
d241e6632669 lfs: fix the inferred remote store path when using a --prefix
Matt Harbison <matt_harbison@yahoo.com>
parents: 37565
diff changeset
774 defaulturl.path = (defaulturl.path or b'') + b'.git/info/lfs'
37518
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
775
46907
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46113
diff changeset
776 url = urlutil.url(bytes(defaulturl))
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
777 repo.ui.note(_(b'lfs: assuming remote store: %s\n') % url)
37518
092eff6833a7 lfs: infer the blob store URL from paths.default
Matt Harbison <matt_harbison@yahoo.com>
parents: 37517
diff changeset
778
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
779 scheme = url.scheme
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
780 if scheme not in _storemap:
41426
02d0a7774882 py3: byteify the LFS blobstore module
Matt Harbison <matt_harbison@yahoo.com>
parents: 40675
diff changeset
781 raise error.Abort(_(b'lfs: unknown url scheme: %s') % scheme)
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
782 return _storemap[scheme](repo, url)
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
783
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
784
39777
b63dee7bd0d9 global: replace most uses of RevlogError with StorageError (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39457
diff changeset
785 class LfsRemoteError(error.StorageError):
35098
66c5a8cf2868 lfs: import the Facebook git-lfs client extension
Matt Harbison <matt_harbison@yahoo.com>
parents:
diff changeset
786 pass
37692
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37691
diff changeset
787
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 41607
diff changeset
788
37692
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37691
diff changeset
789 class LfsCorruptionError(error.Abort):
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37691
diff changeset
790 """Raised when a corrupt blob is detected, aborting an operation
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37691
diff changeset
791
10e5bb9678f4 lfs: gracefully handle aborts on the server when corrupt blobs are detected
Matt Harbison <matt_harbison@yahoo.com>
parents: 37691
diff changeset
792 It exists to allow specialized handling on the server side."""