mercurial-scm/hg-stable: mercurial/bundle2.py annotate

annotate mercurial/bundle2.py @ 20949:571f2903ff1e

bundle2: record processing results in the bundleoperation object Part handlers can now add records to the `bundleoperation` object. This can be used to help other parts or to let the caller of the unbundling process react to the results.

author	Pierre-Yves David <pierre-yves.david@fb.com>
date	Wed, 02 Apr 2014 22:37:50 -0700
parents	329cd74b52bd
children	c7ceae0faf69

rev	line source
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	1 # bundle2.py - generic container format to transmit arbitrary data.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	2 #
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	3 # Copyright 2013 Facebook, Inc.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	4 #
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	7 """Handling of the new bundle2 format
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	8
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	9 The goal of bundle2 is to act as an atomically packet to transmit a set of
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	10 payloads in an application agnostic way. It consist in a sequence of "parts"
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	11 that will be handed to and processed by the application layer.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	12
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	13
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	14 General format architecture
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	15 ===========================
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	16
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	17 The format is architectured as follow
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	18
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	19 - magic string
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	20 - stream level parameters
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	21 - payload parts (any number)
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	22 - end of stream marker.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	23
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	24 the Binary format
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	25 ============================
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	26
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	27 All numbers are unsigned and big endian.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	28
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	29 stream level parameters
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	30 ------------------------
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	31
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	32 Binary format is as follow
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	33
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	34 :params size: (16 bits integer)
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	35
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	36 The total number of Bytes used by the parameters
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	37
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	38 :params value: arbitrary number of Bytes
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	39
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	40 A blob of `params size` containing the serialized version of all stream level
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	41 parameters.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	42
20809 b93bb639451a bundle2: support for bundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20808 diff changeset	43 The blob contains a space separated list of parameters. parameter with value
20811 9785c3f8f598 bundle2: urlquote stream parameter name and value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20810 diff changeset	44 are stored in the form `<name>=<value>`. Both name and value are urlquoted.
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	45
20813 8c74b3ce5b70 bundle2: refuse empty parameter name Pierre-Yves David <pierre-yves.david@fb.com> parents: 20812 diff changeset	46 Empty name are obviously forbidden.
8c74b3ce5b70 bundle2: refuse empty parameter name Pierre-Yves David <pierre-yves.david@fb.com> parents: 20812 diff changeset	47
20844 2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	48 Name MUST start with a letter. If this first letter is lower case, the
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	49 parameter is advisory and can be safefly ignored. However when the first
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	50 letter is capital, the parameter is mandatory and the bundling process MUST
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	51 stop if he is not able to proceed it.
20814 8532f5e1b9df bundle2: force the first char of parameter to be an letter. Pierre-Yves David <pierre-yves.david@fb.com> parents: 20813 diff changeset	52
20808 4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	53 Stream parameters use a simple textual format for two main reasons:
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	54
20808 4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	55 - Stream level parameters should remains simple and we want to discourage any
4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	56 crazy usage.
4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	57 - Textual data allow easy human inspection of a the bundle2 header in case of
4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	58 troubles.
4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	59
4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	60 Any Applicative level options MUST go into a bundle2 part instead.
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	61
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	62 Payload part
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	63 ------------------------
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	64
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	65 Binary format is as follow
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	66
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	67 :header size: (16 bits inter)
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	68
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	69 The total number of Bytes used by the part headers. When the header is empty
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	70 (size = 0) this is interpreted as the end of stream marker.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	71
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	72 :header:
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	73
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	74 The header defines how to interpret the part. It contains two piece of
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	75 data: the part type, and the part parameters.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	76
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	77 The part type is used to route an application level handler, that can
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	78 interpret payload.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	79
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	80 Part parameters are passed to the application level handler. They are
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	81 meant to convey information that will help the application level object to
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	82 interpret the part payload.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	83
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	84 The binary format of the header is has follow
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	85
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	86 :typesize: (one byte)
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	87
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	88 :typename: alphanumerical part name
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	89
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	90 :parameters:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	91
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	92 Part's parameter may have arbitraty content, the binary structure is::
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	93
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	94 <mandatory-count><advisory-count><param-sizes><param-data>
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	95
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	96 :mandatory-count: 1 byte, number of mandatory parameters
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	97
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	98 :advisory-count: 1 byte, number of advisory parameters
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	99
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	100 :param-sizes:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	101
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	102 N couple of bytes, where N is the total number of parameters. Each
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	103 couple contains (<size-of-key>, <size-of-value) for one parameter.
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	104
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	105 :param-data:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	106
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	107 A blob of bytes from which each parameter key and value can be
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	108 retrieved using the list of size couples stored in the previous
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	109 field.
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	110
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	111 Mandatory parameters comes first, then the advisory ones.
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	112
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	113 :payload:
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	114
20876 ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	115 payload is a series of `<chunksize><chunkdata>`.
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	116
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	117 `chunksize` is a 32 bits integer, `chunkdata` are plain bytes (as much as
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	118 `chunksize` says)` The payload part is concluded by a zero size chunk.
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	119
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	120 The current implementation always produces either zero or one chunk.
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	121 This is an implementation limitation that will ultimatly be lifted.
20891 1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	122
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	123 Bundle processing
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	124 ============================
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	125
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	126 Each part is processed in order using a "part handler". Handler are registered
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	127 for a certain part type.
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	128
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	129 The matching of a part to its handler is case insensitive. The case of the
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	130 part type is used to know if a part is mandatory or advisory. If the Part type
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	131 contains any uppercase char it is considered mandatory. When no handler is
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	132 known for a Mandatory part, the process is aborted and an exception is raised.
20892 6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	133 If the part is advisory and no handler is known, the part is ignored. When the
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	134 process is aborted, the full bundle is still read from the stream to keep the
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	135 channel usable. But none of the part read from an abort are processed. In the
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	136 future, dropping the stream may become an option for channel we do not care to
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	137 preserve.
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	138 """
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	139
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	140 import util
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	141 import struct
20811 9785c3f8f598 bundle2: urlquote stream parameter name and value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20810 diff changeset	142 import urllib
20814 8532f5e1b9df bundle2: force the first char of parameter to be an letter. Pierre-Yves David <pierre-yves.david@fb.com> parents: 20813 diff changeset	143 import string
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	144
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	145 import changegroup
20803 88db3e615319 bundle2: make sure the unbundler refuse non bundle2 stream Pierre-Yves David <pierre-yves.david@fb.com> parents: 20802 diff changeset	146 from i18n import _
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	147
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	148 _pack = struct.pack
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	149 _unpack = struct.unpack
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	150
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	151 _magicstring = 'HG20'
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	152
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	153 _fstreamparamsize = '>H'
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	154 _fpartheadersize = '>H'
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	155 _fparttypesize = '>B'
20876 ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	156 _fpayloadsize = '>I'
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	157 _fpartparamcount = '>BB'
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	158
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	159 def _makefpartparamsizes(nbparams):
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	160 """return a struct format to read part parameter sizes
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	161
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	162 The number parameters is variable so we need to build that format
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	163 dynamically.
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	164 """
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	165 return '>'+('BB'*nbparams)
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	166
20890 ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	167 parthandlermapping = {}
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	168
20890 ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	169 def parthandler(parttype):
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	170 """decorator that register a function as a bundle2 part handler
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	171
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	172 eg::
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	173
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	174 @parthandler('myparttype')
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	175 def myparttypehandler(...):
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	176 '''process a part of type "my part".'''
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	177 ...
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	178 """
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	179 def _decorator(func):
20891 1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	180 lparttype = parttype.lower() # enforce lower case matching.
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	181 assert lparttype not in parthandlermapping
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	182 parthandlermapping[lparttype] = func
20890 ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	183 return func
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	184 return _decorator
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	185
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	186 class unbundlerecords(object):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	187 """keep record of what happens during and unbundle
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	188
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	189 New records are added using `records.add('cat', obj)`. Where 'cat' is a
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	190 category of record and obj is an arbitraty object.
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	191
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	192 `records['cat']` will return all entries of this category 'cat'.
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	193
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	194 Iterating on the object itself will yield `('category', obj)` tuples
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	195 for all entries.
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	196
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	197 All iterations happens in chronological order.
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	198 """
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	199
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	200 def __init__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	201 self._categories = {}
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	202 self._sequences = []
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	203
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	204 def add(self, category, entry):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	205 """add a new record of a given category.
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	206
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	207 The entry can then be retrieved in the list returned by
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	208 self['category']."""
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	209 self._categories.setdefault(category, []).append(entry)
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	210 self._sequences.append((category, entry))
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	211
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	212 def __getitem__(self, cat):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	213 return tuple(self._categories.get(cat, ()))
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	214
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	215 def __iter__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	216 return iter(self._sequences)
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	217
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	218 def __len__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	219 return len(self._sequences)
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	220
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	221 def __nonzero__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	222 return bool(self._sequences)
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	223
20948 329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	224 class bundleoperation(object):
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	225 """an object that represents a single bundling process
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	226
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	227 Its purpose is to carry unbundle-related objects and states.
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	228
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	229 A new object should be created at the beginning of each bundle processing.
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	230 The object is to be returned by the processing function.
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	231
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	232 The object has very little content now it will ultimately contain:
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	233 * an access to the repo the bundle is applied to,
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	234 * a ui object,
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	235 * a way to retrieve a transaction to add changes to the repo,
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	236 * a way to record the result of processing each part,
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	237 * a way to construct a bundle response when applicable.
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	238 """
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	239
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	240 def __init__(self, repo):
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	241 self.repo = repo
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	242 self.ui = repo.ui
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	243 self.records = unbundlerecords()
20948 329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	244
20947 c33d7bf53812 bundle2: feed a unbundle20 to the `processbundle` function Pierre-Yves David <pierre-yves.david@fb.com> parents: 20892 diff changeset	245 def processbundle(repo, unbundler):
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	246 """This function process a bundle, apply effect to/from a repo
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	247
20947 c33d7bf53812 bundle2: feed a unbundle20 to the `processbundle` function Pierre-Yves David <pierre-yves.david@fb.com> parents: 20892 diff changeset	248 It iterates over each part then searches for and uses the proper handling
c33d7bf53812 bundle2: feed a unbundle20 to the `processbundle` function Pierre-Yves David <pierre-yves.david@fb.com> parents: 20892 diff changeset	249 code to process the part. Parts are processed in order.
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	250
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	251 This is very early version of this function that will be strongly reworked
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	252 before final usage.
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	253
20891 1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	254 Unknown Mandatory part will abort the process.
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	255 """
20948 329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	256 op = bundleoperation(repo)
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	257 # todo:
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	258 # - replace this is a init function soon.
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	259 # - exception catching
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	260 unbundler.params
20892 6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	261 iterparts = iter(unbundler)
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	262 try:
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	263 for part in iterparts:
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	264 parttype = part.type
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	265 # part key are matched lower case
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	266 key = parttype.lower()
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	267 try:
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	268 handler = parthandlermapping[key]
20948 329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	269 op.ui.debug('found a handler for part %r\n' % parttype)
20892 6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	270 except KeyError:
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	271 if key != parttype: # mandatory parts
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	272 # todo:
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	273 # - use a more precise exception
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	274 raise
20948 329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	275 op.ui.debug('ignoring unknown advisory part %r\n' % key)
20891 1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	276 # todo:
20892 6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	277 # - consume the part once we use streaming
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	278 continue
20948 329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	279 handler(op, part)
20892 6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	280 except Exception:
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	281 for part in iterparts:
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	282 pass # consume the bundle content
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	283 raise
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	284 return op
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	285
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	286 class bundle20(object):
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	287 """represent an outgoing bundle2 container
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	288
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	289 Use the `addparam` method to add stream level parameter. and `addpart` to
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	290 populate it. Then call `getchunks` to retrieve all the binary chunks of
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	291 datathat compose the bundle2 container."""
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	292
20842 938718d72624 bundle2: print debug information during bundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20814 diff changeset	293 def __init__(self, ui):
938718d72624 bundle2: print debug information during bundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20814 diff changeset	294 self.ui = ui
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	295 self._params = []
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	296 self._parts = []
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	297
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	298 def addparam(self, name, value=None):
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	299 """add a stream level parameter"""
20813 8c74b3ce5b70 bundle2: refuse empty parameter name Pierre-Yves David <pierre-yves.david@fb.com> parents: 20812 diff changeset	300 if not name:
8c74b3ce5b70 bundle2: refuse empty parameter name Pierre-Yves David <pierre-yves.david@fb.com> parents: 20812 diff changeset	301 raise ValueError('empty parameter name')
20814 8532f5e1b9df bundle2: force the first char of parameter to be an letter. Pierre-Yves David <pierre-yves.david@fb.com> parents: 20813 diff changeset	302 if name[0] not in string.letters:
8532f5e1b9df bundle2: force the first char of parameter to be an letter. Pierre-Yves David <pierre-yves.david@fb.com> parents: 20813 diff changeset	303 raise ValueError('non letter first character: %r' % name)
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	304 self._params.append((name, value))
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	305
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	306 def addpart(self, part):
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	307 """add a new part to the bundle2 container
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	308
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	309 Parts contains the actuall applicative payload."""
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	310 self._parts.append(part)
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	311
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	312 def getchunks(self):
20842 938718d72624 bundle2: print debug information during bundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20814 diff changeset	313 self.ui.debug('start emission of %s stream\n' % _magicstring)
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	314 yield _magicstring
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	315 param = self._paramchunk()
20842 938718d72624 bundle2: print debug information during bundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20814 diff changeset	316 self.ui.debug('bundle parameter: %s\n' % param)
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	317 yield _pack(_fstreamparamsize, len(param))
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	318 if param:
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	319 yield param
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	320
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	321 self.ui.debug('start of parts\n')
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	322 for part in self._parts:
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	323 self.ui.debug('bundle part: "%s"\n' % part.type)
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	324 for chunk in part.getchunks():
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	325 yield chunk
20842 938718d72624 bundle2: print debug information during bundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20814 diff changeset	326 self.ui.debug('end of bundle\n')
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	327 yield '\0\0'
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	328
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	329 def _paramchunk(self):
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	330 """return a encoded version of all stream parameters"""
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	331 blocks = []
20809 b93bb639451a bundle2: support for bundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20808 diff changeset	332 for par, value in self._params:
20811 9785c3f8f598 bundle2: urlquote stream parameter name and value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20810 diff changeset	333 par = urllib.quote(par)
20809 b93bb639451a bundle2: support for bundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20808 diff changeset	334 if value is not None:
20811 9785c3f8f598 bundle2: urlquote stream parameter name and value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20810 diff changeset	335 value = urllib.quote(value)
20809 b93bb639451a bundle2: support for bundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20808 diff changeset	336 par = '%s=%s' % (par, value)
b93bb639451a bundle2: support for bundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20808 diff changeset	337 blocks.append(par)
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	338 return ' '.join(blocks)
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	339
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	340 class unbundle20(object):
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	341 """interpret a bundle2 stream
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	342
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	343 (this will eventually yield parts)"""
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	344
20843 0641b41b0b49 bundle2: print debug information during unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20842 diff changeset	345 def __init__(self, ui, fp):
0641b41b0b49 bundle2: print debug information during unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20842 diff changeset	346 self.ui = ui
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	347 self._fp = fp
20803 88db3e615319 bundle2: make sure the unbundler refuse non bundle2 stream Pierre-Yves David <pierre-yves.david@fb.com> parents: 20802 diff changeset	348 header = self._readexact(4)
88db3e615319 bundle2: make sure the unbundler refuse non bundle2 stream Pierre-Yves David <pierre-yves.david@fb.com> parents: 20802 diff changeset	349 magic, version = header[0:2], header[2:4]
88db3e615319 bundle2: make sure the unbundler refuse non bundle2 stream Pierre-Yves David <pierre-yves.david@fb.com> parents: 20802 diff changeset	350 if magic != 'HG':
88db3e615319 bundle2: make sure the unbundler refuse non bundle2 stream Pierre-Yves David <pierre-yves.david@fb.com> parents: 20802 diff changeset	351 raise util.Abort(_('not a Mercurial bundle'))
88db3e615319 bundle2: make sure the unbundler refuse non bundle2 stream Pierre-Yves David <pierre-yves.david@fb.com> parents: 20802 diff changeset	352 if version != '20':
88db3e615319 bundle2: make sure the unbundler refuse non bundle2 stream Pierre-Yves David <pierre-yves.david@fb.com> parents: 20802 diff changeset	353 raise util.Abort(_('unknown bundle version %s') % version)
20843 0641b41b0b49 bundle2: print debug information during unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20842 diff changeset	354 self.ui.debug('start processing of %s stream\n' % header)
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	355
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	356 def _unpack(self, format):
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	357 """unpack this struct format from the stream"""
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	358 data = self._readexact(struct.calcsize(format))
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	359 return _unpack(format, data)
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	360
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	361 def _readexact(self, size):
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	362 """read exactly <size> bytes from the stream"""
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	363 return changegroup.readexactly(self._fp, size)
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	364
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	365 @util.propertycache
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	366 def params(self):
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	367 """dictionnary of stream level parameters"""
20843 0641b41b0b49 bundle2: print debug information during unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20842 diff changeset	368 self.ui.debug('reading bundle2 stream parameters\n')
20805 c5aaeca0cfbf bundle2: support for unbundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20804 diff changeset	369 params = {}
c5aaeca0cfbf bundle2: support for unbundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20804 diff changeset	370 paramssize = self._unpack(_fstreamparamsize)[0]
c5aaeca0cfbf bundle2: support for unbundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20804 diff changeset	371 if paramssize:
c5aaeca0cfbf bundle2: support for unbundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20804 diff changeset	372 for p in self._readexact(paramssize).split(' '):
20810 47293877b54c bundle2: support for unbundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20809 diff changeset	373 p = p.split('=', 1)
20812 e2f908773754 bundle2: urlunquote stream parameter name and value during unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20811 diff changeset	374 p = [urllib.unquote(i) for i in p]
20810 47293877b54c bundle2: support for unbundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20809 diff changeset	375 if len(p) < 2:
47293877b54c bundle2: support for unbundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20809 diff changeset	376 p.append(None)
20844 2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	377 self._processparam(*p)
20810 47293877b54c bundle2: support for unbundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20809 diff changeset	378 params[p[0]] = p[1]
20805 c5aaeca0cfbf bundle2: support for unbundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20804 diff changeset	379 return params
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	380
20844 2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	381 def _processparam(self, name, value):
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	382 """process a parameter, applying its effect if needed
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	383
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	384 Parameter starting with a lower case letter are advisory and will be
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	385 ignored when unknown. Those starting with an upper case letter are
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	386 mandatory and will this function will raise a KeyError when unknown.
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	387
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	388 Note: no option are currently supported. Any input will be either
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	389 ignored or failing.
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	390 """
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	391 if not name:
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	392 raise ValueError('empty parameter name')
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	393 if name[0] not in string.letters:
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	394 raise ValueError('non letter first character: %r' % name)
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	395 # Some logic will be later added here to try to process the option for
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	396 # a dict of known parameter.
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	397 if name[0].islower():
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	398 self.ui.debug("ignoring unknown parameter %r\n" % name)
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	399 else:
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	400 raise KeyError(name)
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	401
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	402
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	403 def __iter__(self):
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	404 """yield all parts contained in the stream"""
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	405 # make sure param have been loaded
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	406 self.params
20843 0641b41b0b49 bundle2: print debug information during unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20842 diff changeset	407 self.ui.debug('start extraction of bundle2 parts\n')
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	408 part = self._readpart()
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	409 while part is not None:
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	410 yield part
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	411 part = self._readpart()
20843 0641b41b0b49 bundle2: print debug information during unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20842 diff changeset	412 self.ui.debug('end of bundle2 stream\n')
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	413
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	414 def _readpart(self):
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	415 """return None when an end of stream markers is reach"""
20864 9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	416
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	417 headersize = self._unpack(_fpartheadersize)[0]
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	418 self.ui.debug('part header size: %i\n' % headersize)
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	419 if not headersize:
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	420 return None
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	421 headerblock = self._readexact(headersize)
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	422 # some utility to help reading from the header block
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	423 self._offset = 0 # layer violation to have something easy to understand
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	424 def fromheader(size):
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	425 """return the next <size> byte from the header"""
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	426 offset = self._offset
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	427 data = headerblock[offset:(offset + size)]
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	428 self._offset = offset + size
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	429 return data
20887 662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	430 def unpackheader(format):
662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	431 """read given format from header
662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	432
662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	433 This automatically compute the size of the format to read."""
662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	434 data = fromheader(struct.calcsize(format))
662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	435 return _unpack(format, data)
662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	436
662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	437 typesize = unpackheader(_fparttypesize)[0]
20864 9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	438 parttype = fromheader(typesize)
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	439 self.ui.debug('part type: "%s"\n' % parttype)
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	440 ## reading parameters
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	441 # param count
20887 662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	442 mancount, advcount = unpackheader(_fpartparamcount)
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	443 self.ui.debug('part parameters: %i\n' % (mancount + advcount))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	444 # param size
20887 662b79be093c bundle2: safely read unpack data from part header Pierre-Yves David <pierre-yves.david@fb.com> parents: 20877 diff changeset	445 paramsizes = unpackheader(_makefpartparamsizes(mancount + advcount))
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	446 # make it a list of couple again
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	447 paramsizes = zip(paramsizes[::2], paramsizes[1::2])
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	448 # split mandatory from advisory
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	449 mansizes = paramsizes[:mancount]
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	450 advsizes = paramsizes[mancount:]
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	451 # retrive param value
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	452 manparams = []
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	453 for key, value in mansizes:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	454 manparams.append((fromheader(key), fromheader(value)))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	455 advparams = []
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	456 for key, value in advsizes:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	457 advparams.append((fromheader(key), fromheader(value)))
20864 9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	458 del self._offset # clean up layer, nobody saw anything.
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	459 ## part payload
20876 ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	460 payload = []
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	461 payloadsize = self._unpack(_fpayloadsize)[0]
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	462 self.ui.debug('payload chunk size: %i\n' % payloadsize)
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	463 while payloadsize:
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	464 payload.append(self._readexact(payloadsize))
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	465 payloadsize = self._unpack(_fpayloadsize)[0]
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	466 self.ui.debug('payload chunk size: %i\n' % payloadsize)
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	467 payload = ''.join(payload)
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	468 current = part(parttype, manparams, advparams, data=payload)
20864 9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	469 return current
9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	470
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	471
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	472 class part(object):
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	473 """A bundle2 part contains application level payload
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	474
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	475 The part `type` is used to route the part to the application level
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	476 handler.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	477 """
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	478
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	479 def __init__(self, parttype, mandatoryparams=(), advisoryparams=(),
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	480 data=''):
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	481 self.type = parttype
20864 9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	482 self.data = data
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	483 self.mandatoryparams = mandatoryparams
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	484 self.advisoryparams = advisoryparams
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	485
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	486 def getchunks(self):
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	487 #### header
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	488 ## parttype
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	489 header = [_pack(_fparttypesize, len(self.type)),
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	490 self.type,
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	491 ]
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	492 ## parameters
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	493 # count
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	494 manpar = self.mandatoryparams
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	495 advpar = self.advisoryparams
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	496 header.append(_pack(_fpartparamcount, len(manpar), len(advpar)))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	497 # size
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	498 parsizes = []
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	499 for key, value in manpar:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	500 parsizes.append(len(key))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	501 parsizes.append(len(value))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	502 for key, value in advpar:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	503 parsizes.append(len(key))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	504 parsizes.append(len(value))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	505 paramsizes = _pack(_makefpartparamsizes(len(parsizes) / 2), *parsizes)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	506 header.append(paramsizes)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	507 # key, value
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	508 for key, value in manpar:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	509 header.append(key)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	510 header.append(value)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	511 for key, value in advpar:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	512 header.append(key)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	513 header.append(value)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	514 ## finalize header
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	515 headerchunk = ''.join(header)
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	516 yield _pack(_fpartheadersize, len(headerchunk))
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	517 yield headerchunk
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	518 ## payload
20876 ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	519 # we only support fixed size data now.
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	520 # This will be improved in the future.
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	521 if len(self.data):
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	522 yield _pack(_fpayloadsize, len(self.data))
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	523 yield self.data
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	524 # end of payload
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	525 yield _pack(_fpayloadsize, 0)
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	526

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/bundle2.py @ 20949:571f2903ff1e