Mercurial > public > mercurial-scm > hg

comparison hgext/lfs/blobstore.py @ 44086:ffac09da7a19

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
lfs: avoid quadratic performance in processing server responses This is also adapted from the Facebook repo[1]. Unlike there, we were already reading the download stream in chunks and immediately writing it to disk, so we basically avoided the problem on download. There shouldn't be a lot of data to read on upload, but it's better to get rid of this pattern. [1] https://github.com/facebookexperimental/eden/commit/82df66ffe97e21f3ee73dfec093c87500fc1f6a7 Differential Revision: https://phab.mercurial-scm.org/D7882
author Matt Harbison <matt_harbison@yahoo.com>
date Tue, 14 Jan 2020 20:05:37 -0500
parents 0ee0a3f6a990
children 5f841daf3b41
comparison
equal deleted inserted replaced
44085:0ee0a3f6a990 44086:ffac09da7a19
501 request.add_header('Content-Length', len(request.data)) 501 request.add_header('Content-Length', len(request.data))
502 502
503 for k, v in headers: 503 for k, v in headers:
504 request.add_header(pycompat.strurl(k), pycompat.strurl(v)) 504 request.add_header(pycompat.strurl(k), pycompat.strurl(v))
505 505
506 response = b''
507 try: 506 try:
508 with contextlib.closing(self.urlopener.open(request)) as res: 507 with contextlib.closing(self.urlopener.open(request)) as res:
509 contentlength = res.info().get(b"content-length") 508 contentlength = res.info().get(b"content-length")
510 ui = self.ui # Shorten debug lines 509 ui = self.ui # Shorten debug lines
511 if self.ui.debugflag: 510 if self.ui.debugflag:
518 if action == b'download': 517 if action == b'download':
519 # If downloading blobs, store downloaded data to local 518 # If downloading blobs, store downloaded data to local
520 # blobstore 519 # blobstore
521 localstore.download(oid, res, contentlength) 520 localstore.download(oid, res, contentlength)
522 else: 521 else:
522 blocks = []
523 while True: 523 while True:
524 data = res.read(1048576) 524 data = res.read(1048576)
525 if not data: 525 if not data:
526 break 526 break
527 response += data 527 blocks.append(data)
528
529 response = b"".join(blocks)
528 if response: 530 if response:
529 ui.debug(b'lfs %s response: %s' % (action, response)) 531 ui.debug(b'lfs %s response: %s' % (action, response))
530 except util.urlerr.httperror as ex: 532 except util.urlerr.httperror as ex:
531 if self.ui.debugflag: 533 if self.ui.debugflag:
532 self.ui.debug( 534 self.ui.debug(