Mercurial > public > mercurial-scm > hg

comparison mercurial/wireprotov2server.py @ 39641:aa7e312375cf

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
wireprotov2: let clients drive delta behavior Previously, the "manifestdata" and "filedata" commands assumed the receiver had all parent revisions for requested nodes. Unless the revision had no parents, they emitted a delta instead of a fulltext. This strategy isn't appropriate for shallow clones and for clients that only want to access fulltext revision data for a single node without fetching their parent revisions. This commit adds an "haveparents" argument to the "manifestdata" and "filedata" commands that controls delta generation behavior. Unless "haveparents" is set, the server assumes that the client doesn't have parent revisions unless they were previously sent as part of the current group of revisions. This change allows the fulltext revision data of any individual revision to be obtained. This will facilitate shallow clones and other data retrieval strategies that don't require all previous revisions of an entity to be fetched. Differential Revision: https://phab.mercurial-scm.org/D4492
author Gregory Szorc <gregory.szorc@gmail.com>
date Thu, 30 Aug 2018 14:55:34 -0700
parents 0e03e6a44dee
children 0b61d21f05cc
comparison
equal deleted inserted replaced
39640:039bf1eddc2e 39641:aa7e312375cf
413 caps['rawrepoformats'] = sorted(repo.requirements & 413 caps['rawrepoformats'] = sorted(repo.requirements &
414 repo.supportedformats) 414 repo.supportedformats)
415 415
416 return proto.addcapabilities(repo, caps) 416 return proto.addcapabilities(repo, caps)
417 417
418 def builddeltarequests(store, nodes): 418 def builddeltarequests(store, nodes, haveparents):
419 """Build a series of revision delta requests against a backend store. 419 """Build a series of revision delta requests against a backend store.
420 420
421 Returns a list of revision numbers in the order they should be sent 421 Returns a list of revision numbers in the order they should be sent
422 and a list of ``irevisiondeltarequest`` instances to be made against 422 and a list of ``irevisiondeltarequest`` instances to be made against
423 the backend store. 423 the backend store.
428 # a list of nodes and delta preconditions over a figurative wall and 428 # a list of nodes and delta preconditions over a figurative wall and
429 # have the storage backend figure it out for us. 429 # have the storage backend figure it out for us.
430 revs = dagop.linearize({store.rev(n) for n in nodes}, store.parentrevs) 430 revs = dagop.linearize({store.rev(n) for n in nodes}, store.parentrevs)
431 431
432 requests = [] 432 requests = []
433 seenrevs = set()
433 434
434 for rev in revs: 435 for rev in revs:
435 node = store.node(rev) 436 node = store.node(rev)
436 parents = store.parents(node) 437 parentnodes = store.parents(node)
437 deltaparent = store.node(store.deltaparent(rev)) 438 parentrevs = [store.rev(n) for n in parentnodes]
438 439 deltabaserev = store.deltaparent(rev)
439 # There is a delta in storage. That means we can send the delta 440 deltabasenode = store.node(deltabaserev)
440 # efficiently. 441
442 # The choice of whether to send a fulltext revision or a delta and
443 # what delta to send is governed by a few factors.
441 # 444 #
442 # But, the delta may be against a revision the receiver doesn't 445 # To send a delta, we need to ensure the receiver is capable of
443 # have (e.g. shallow clone or when the delta isn't against a parent 446 # decoding it. And that requires the receiver to have the base
444 # revision). For now, we ignore the problem of shallow clone. As 447 # revision the delta is against.
445 # long as a delta exists against a parent, we send it.
446 # TODO allow arguments to control this behavior, as the receiver
447 # may not have the base revision in some scenarios.
448 if deltaparent != nullid and deltaparent in parents:
449 basenode = deltaparent
450
451 # Else there is no delta parent in storage or the delta that is
452 # # there isn't suitable. Let's use a delta against a parent
453 # revision, if possible.
454 # 448 #
455 # There is room to check if the delta parent is in the ancestry of 449 # We can only guarantee the receiver has the base revision if
456 # this node. But there isn't an API on the manifest storage object 450 # a) we've already sent the revision as part of this group
457 # for that. So ignore this case for now. 451 # b) the receiver has indicated they already have the revision.
458 452 # And the mechanism for "b" is the client indicating they have
459 elif parents[0] != nullid: 453 # parent revisions. So this means we can only send the delta if
460 basenode = parents[0] 454 # it is sent before or it is against a delta and the receiver says
461 elif parents[1] != nullid: 455 # they have a parent.
462 basenode = parents[1] 456
463 457 # We can send storage delta if it is against a revision we've sent
464 # No potential bases to delta against. Send a full revision. 458 # in this group.
459 if deltabaserev != nullrev and deltabaserev in seenrevs:
460 basenode = deltabasenode
461
462 # We can send storage delta if it is against a parent revision and
463 # the receiver indicates they have the parents.
464 elif (deltabaserev != nullrev and deltabaserev in parentrevs
465 and haveparents):
466 basenode = deltabasenode
467
468 # Otherwise the storage delta isn't appropriate. Fall back to
469 # using another delta, if possible.
470
471 # Use p1 if we've emitted it or receiver says they have it.
472 elif parentrevs[0] != nullrev and (
473 parentrevs[0] in seenrevs or haveparents):
474 basenode = parentnodes[0]
475
476 # Use p2 if we've emitted it or receiver says they have it.
477 elif parentrevs[1] != nullrev and (
478 parentrevs[1] in seenrevs or haveparents):
479 basenode = parentnodes[1]
480
481 # Nothing appropriate to delta against. Send the full revision.
465 else: 482 else:
466 basenode = nullid 483 basenode = nullid
467 484
468 requests.append(changegroup.revisiondeltarequest( 485 requests.append(changegroup.revisiondeltarequest(
469 node=node, 486 node=node,
470 p1node=parents[0], 487 p1node=parentnodes[0],
471 p2node=parents[1], 488 p2node=parentnodes[1],
472 # Receiver deals with linknode resolution. 489 # Receiver deals with linknode resolution.
473 linknode=nullid, 490 linknode=nullid,
474 basenode=basenode, 491 basenode=basenode,
475 )) 492 ))
493
494 seenrevs.add(rev)
476 495
477 return revs, requests 496 return revs, requests
478 497
479 def wireprotocommand(name, args=None, permission='push'): 498 def wireprotocommand(name, args=None, permission='push'):
480 """Decorator to declare a wire protocol command. 499 """Decorator to declare a wire protocol command.
672 691
673 return fl 692 return fl
674 693
675 @wireprotocommand('filedata', 694 @wireprotocommand('filedata',
676 args={ 695 args={
696 'haveparents': True,
677 'nodes': [b'0123456...'], 697 'nodes': [b'0123456...'],
678 'fields': [b'parents', b'revision'], 698 'fields': [b'parents', b'revision'],
679 'path': b'foo.txt', 699 'path': b'foo.txt',
680 }, 700 },
681 permission='pull') 701 permission='pull')
682 def filedata(repo, proto, nodes=None, fields=None, path=None): 702 def filedata(repo, proto, haveparents=False, nodes=None, fields=None,
703 path=None):
683 fields = fields or set() 704 fields = fields or set()
684 705
685 if nodes is None: 706 if nodes is None:
686 raise error.WireprotoCommandError('nodes argument must be defined') 707 raise error.WireprotoCommandError('nodes argument must be defined')
687 708
700 store.rev(node) 721 store.rev(node)
701 except error.LookupError: 722 except error.LookupError:
702 raise error.WireprotoCommandError('unknown file node: %s', 723 raise error.WireprotoCommandError('unknown file node: %s',
703 (hex(node),)) 724 (hex(node),))
704 725
705 revs, requests = builddeltarequests(store, nodes) 726 revs, requests = builddeltarequests(store, nodes, haveparents)
706 727
707 yield { 728 yield {
708 b'totalitems': len(revs), 729 b'totalitems': len(revs),
709 } 730 }
710 731
802 yield node 823 yield node
803 824
804 @wireprotocommand('manifestdata', 825 @wireprotocommand('manifestdata',
805 args={ 826 args={
806 'nodes': [b'0123456...'], 827 'nodes': [b'0123456...'],
828 'haveparents': True,
807 'fields': [b'parents', b'revision'], 829 'fields': [b'parents', b'revision'],
808 'tree': b'', 830 'tree': b'',
809 }, 831 },
810 permission='pull') 832 permission='pull')
811 def manifestdata(repo, proto, nodes=None, fields=None, tree=None): 833 def manifestdata(repo, proto, haveparents=False, nodes=None, fields=None,
834 tree=None):
812 fields = fields or set() 835 fields = fields or set()
813 836
814 if nodes is None: 837 if nodes is None:
815 raise error.WireprotoCommandError( 838 raise error.WireprotoCommandError(
816 'nodes argument must be defined') 839 'nodes argument must be defined')
827 store.rev(node) 850 store.rev(node)
828 except error.LookupError: 851 except error.LookupError:
829 raise error.WireprotoCommandError( 852 raise error.WireprotoCommandError(
830 'unknown node: %s', (node,)) 853 'unknown node: %s', (node,))
831 854
832 revs, requests = builddeltarequests(store, nodes) 855 revs, requests = builddeltarequests(store, nodes, haveparents)
833 856
834 yield { 857 yield {
835 b'totalitems': len(revs), 858 b'totalitems': len(revs),
836 } 859 }
837 860