comparison mercurial/exchange.py @ 35787:a84dbc87dae9

exchange: send bundle2 stream clones uncompressed Stream clones don't compress well. And compression undermines a point of stream clones which is to trade significant CPU reductions by increasing size. Building upon our introduction of metadata to communicate bundle information back to callers of exchange.getbundlechunks(), we add an attribute to the bundler that communicates whether the bundle is best left uncompressed. We return this attribute as part of the bundle metadata. And the wire protocol honors it when determining whether to compress the wire protocol response. The added test demonstrates that the raw result from the wire protocol is not compressed. It also demonstrates that the server will serve stream responses when the feature isn't enabled. We'll address that in another commit. The effect of this change is that server-side CPU usage for bundle2 stream clones is significantly reduced by removing zstd compression. For the mozilla-unified repository: before: 37.69 user 8.01 system after: 27.38 user 7.34 system Assuming things are CPU bound, that ~10s reduction would translate to faster clones on the client. zstd can decompress at >1 GB/s. So the overhead from decompression on the client is small in the grand scheme of things. But if zlib compression were being used, the overhead would be much greater. Differential Revision: https://phab.mercurial-scm.org/D1926
author Gregory Szorc <gregory.szorc@gmail.com>
date Mon, 22 Jan 2018 12:12:29 -0800
parents ba15580e53d5
children b116a66bcc44
comparison
equal deleted inserted replaced
35786:8144f1b07e21 35787:a84dbc87dae9
1761 for name in getbundle2partsorder: 1761 for name in getbundle2partsorder:
1762 func = getbundle2partsmapping[name] 1762 func = getbundle2partsmapping[name]
1763 func(bundler, repo, source, bundlecaps=bundlecaps, b2caps=b2caps, 1763 func(bundler, repo, source, bundlecaps=bundlecaps, b2caps=b2caps,
1764 **pycompat.strkwargs(kwargs)) 1764 **pycompat.strkwargs(kwargs))
1765 1765
1766 info['prefercompressed'] = bundler.prefercompressed
1767
1766 return info, bundler.getchunks() 1768 return info, bundler.getchunks()
1767 1769
1768 @getbundle2partsgenerator('stream') 1770 @getbundle2partsgenerator('stream')
1769 def _getbundlestream(bundler, repo, source, bundlecaps=None, 1771 def _getbundlestream(bundler, repo, source, bundlecaps=None,
1770 b2caps=None, heads=None, common=None, **kwargs): 1772 b2caps=None, heads=None, common=None, **kwargs):
1771 if not kwargs.get('stream', False): 1773 if not kwargs.get('stream', False):
1772 return 1774 return
1775
1776 # Stream clones don't compress well. And compression undermines a
1777 # goal of stream clones, which is to be fast. Communicate the desire
1778 # to avoid compression to consumers of the bundle.
1779 bundler.prefercompressed = False
1780
1773 filecount, bytecount, it = streamclone.generatev2(repo) 1781 filecount, bytecount, it = streamclone.generatev2(repo)
1774 requirements = ' '.join(sorted(repo.requirements)) 1782 requirements = ' '.join(sorted(repo.requirements))
1775 part = bundler.newpart('stream', data=it) 1783 part = bundler.newpart('stream', data=it)
1776 part.addparam('bytecount', '%d' % bytecount, mandatory=True) 1784 part.addparam('bytecount', '%d' % bytecount, mandatory=True)
1777 part.addparam('filecount', '%d' % filecount, mandatory=True) 1785 part.addparam('filecount', '%d' % filecount, mandatory=True)