Mercurial > public > mercurial-scm > hg
comparison mercurial/util.py @ 26480:6ae14d1ca3aa
util.chunkbuffer: avoid extra mutations when reading partial chunks
Previously, a read(N) where N was less than the length of the first
available chunk would mutate the deque instance twice and allocate a new
str from the slice of the existing chunk. Profiling drawed my attention
to these as a potential hot spot during changegroup reading.
This patch makes the code more complicated in order to avoid the
aforementioned 3 operations.
On a pre-generated mozilla-central gzip bundle, this series has the
following impact on `hg unbundle` performance on my MacBook Pro:
before: 358.21 real 317.69 user 38.49 sys
after: 301.57 real 262.69 user 37.11 sys
delta: -56.64 real -55.00 user -1.38 sys
| author | Gregory Szorc <gregory.szorc@gmail.com> |
|---|---|
| date | Mon, 05 Oct 2015 17:36:32 -0700 |
| parents | 46143f31290e |
| children | 7d132557e44a |
comparison
equal
deleted
inserted
replaced
| 26479:46143f31290e | 26480:6ae14d1ca3aa |
|---|---|
| 1284 pos = end | 1284 pos = end |
| 1285 else: | 1285 else: |
| 1286 yield chunk | 1286 yield chunk |
| 1287 self.iter = splitbig(in_iter) | 1287 self.iter = splitbig(in_iter) |
| 1288 self._queue = collections.deque() | 1288 self._queue = collections.deque() |
| 1289 self._chunkoffset = 0 | |
| 1289 | 1290 |
| 1290 def read(self, l=None): | 1291 def read(self, l=None): |
| 1291 """Read L bytes of data from the iterator of chunks of data. | 1292 """Read L bytes of data from the iterator of chunks of data. |
| 1292 Returns less than L bytes if the iterator runs dry. | 1293 Returns less than L bytes if the iterator runs dry. |
| 1293 | 1294 |
| 1308 if target <= 0: | 1309 if target <= 0: |
| 1309 break | 1310 break |
| 1310 if not queue: | 1311 if not queue: |
| 1311 break | 1312 break |
| 1312 | 1313 |
| 1314 # The easy way to do this would be to queue.popleft(), modify the | |
| 1315 # chunk (if necessary), then queue.appendleft(). However, for cases | |
| 1316 # where we read partial chunk content, this incurs 2 dequeue | |
| 1317 # mutations and creates a new str for the remaining chunk in the | |
| 1318 # queue. Our code below avoids this overhead. | |
| 1319 | |
| 1313 chunk = queue[0] | 1320 chunk = queue[0] |
| 1314 chunkl = len(chunk) | 1321 chunkl = len(chunk) |
| 1322 offset = self._chunkoffset | |
| 1315 | 1323 |
| 1316 # Use full chunk. | 1324 # Use full chunk. |
| 1317 if left >= chunkl: | 1325 if offset == 0 and left >= chunkl: |
| 1318 left -= chunkl | 1326 left -= chunkl |
| 1319 queue.popleft() | 1327 queue.popleft() |
| 1320 buf.append(chunk) | 1328 buf.append(chunk) |
| 1329 # self._chunkoffset remains at 0. | |
| 1330 continue | |
| 1331 | |
| 1332 chunkremaining = chunkl - offset | |
| 1333 | |
| 1334 # Use all of unconsumed part of chunk. | |
| 1335 if left >= chunkremaining: | |
| 1336 left -= chunkremaining | |
| 1337 queue.popleft() | |
| 1338 # offset == 0 is enabled by block above, so this won't merely | |
| 1339 # copy via ``chunk[0:]``. | |
| 1340 buf.append(chunk[offset:]) | |
| 1341 self._chunkoffset = 0 | |
| 1342 | |
| 1321 # Partial chunk needed. | 1343 # Partial chunk needed. |
| 1322 else: | 1344 else: |
| 1323 left -= chunkl | 1345 buf.append(chunk[offset:offset + left]) |
| 1324 queue.popleft() | 1346 self._chunkoffset += left |
| 1325 queue.appendleft(chunk[left:]) | 1347 left -= chunkremaining |
| 1326 buf.append(chunk[:left]) | |
| 1327 | 1348 |
| 1328 return ''.join(buf) | 1349 return ''.join(buf) |
| 1329 | 1350 |
| 1330 def filechunkiter(f, size=65536, limit=None): | 1351 def filechunkiter(f, size=65536, limit=None): |
| 1331 """Create a generator that produces the data in the file size | 1352 """Create a generator that produces the data in the file size |