Mercurial > public > mercurial-scm > hg
comparison mercurial/revlog.py @ 38643:967fee55e8d9
revlog: postprocess chunk to slice them down to a certain size
After the density slicing is done, we enforce a maximum chunk size to avoid
memory consumption issue.
| author | Boris Feld <boris.feld@octobus.net> |
|---|---|
| date | Tue, 10 Jul 2018 11:57:33 +0200 |
| parents | e59e27e52297 |
| children | 43d0619cec90 |
comparison
equal
deleted
inserted
replaced
| 38642:e59e27e52297 | 38643:967fee55e8d9 |
|---|---|
| 291 """ | 291 """ |
| 292 if not revs: | 292 if not revs: |
| 293 return 0 | 293 return 0 |
| 294 return revlog.end(revs[-1]) - revlog.start(revs[0]) | 294 return revlog.end(revs[-1]) - revlog.start(revs[0]) |
| 295 | 295 |
| 296 def _slicechunk(revlog, revs): | 296 def _slicechunk(revlog, revs, targetsize=None): |
| 297 """slice revs to reduce the amount of unrelated data to be read from disk. | 297 """slice revs to reduce the amount of unrelated data to be read from disk. |
| 298 | 298 |
| 299 ``revs`` is sliced into groups that should be read in one time. | 299 ``revs`` is sliced into groups that should be read in one time. |
| 300 Assume that revs are sorted. | 300 Assume that revs are sorted. |
| 301 | 301 |
| 302 The initial chunk is sliced until the overall density (payload/chunks-span | 302 The initial chunk is sliced until the overall density (payload/chunks-span |
| 303 ratio) is above `revlog._srdensitythreshold`. No gap smaller than | 303 ratio) is above `revlog._srdensitythreshold`. No gap smaller than |
| 304 `revlog._srmingapsize` is skipped. | 304 `revlog._srmingapsize` is skipped. |
| 305 | |
| 306 If `targetsize` is set, no chunk larger than `targetsize` will be yield. | |
| 307 For consistency with other slicing choice, this limit won't go lower than | |
| 308 `revlog._srmingapsize`. | |
| 309 | |
| 310 If individual revisions chunk are larger than this limit, they will still | |
| 311 be raised individually. | |
| 305 | 312 |
| 306 >>> revlog = _testrevlog([ | 313 >>> revlog = _testrevlog([ |
| 307 ... 5, #00 (5) | 314 ... 5, #00 (5) |
| 308 ... 10, #01 (5) | 315 ... 10, #01 (5) |
| 309 ... 12, #02 (2) | 316 ... 12, #02 (2) |
| 330 [[0], [11], [15]] | 337 [[0], [11], [15]] |
| 331 >>> list(_slicechunk(revlog, [0, 11, 13, 15])) | 338 >>> list(_slicechunk(revlog, [0, 11, 13, 15])) |
| 332 [[0], [11, 13, 15]] | 339 [[0], [11, 13, 15]] |
| 333 >>> list(_slicechunk(revlog, [1, 2, 3, 5, 8, 10, 11, 14])) | 340 >>> list(_slicechunk(revlog, [1, 2, 3, 5, 8, 10, 11, 14])) |
| 334 [[1, 2], [5, 8, 10, 11], [14]] | 341 [[1, 2], [5, 8, 10, 11], [14]] |
| 342 | |
| 343 Slicing with a maximum chunk size | |
| 344 >>> list(_slicechunk(revlog, [0, 11, 13, 15], 15)) | |
| 345 [[0], [11], [13], [15]] | |
| 346 >>> list(_slicechunk(revlog, [0, 11, 13, 15], 20)) | |
| 347 [[0], [11], [13, 15]] | |
| 335 """ | 348 """ |
| 349 if targetsize is not None: | |
| 350 targetsize = max(targetsize, revlog._srmingapsize) | |
| 336 for chunk in _slicechunktodensity(revlog, revs, | 351 for chunk in _slicechunktodensity(revlog, revs, |
| 337 revlog._srdensitythreshold, | 352 revlog._srdensitythreshold, |
| 338 revlog._srmingapsize): | 353 revlog._srmingapsize): |
| 339 yield chunk | 354 for subchunk in _slicechunktosize(revlog, chunk, targetsize): |
| 355 yield subchunk | |
| 340 | 356 |
| 341 def _slicechunktosize(revlog, revs, targetsize): | 357 def _slicechunktosize(revlog, revs, targetsize): |
| 342 """slice revs to match the target size | 358 """slice revs to match the target size |
| 343 | 359 |
| 344 This is intended to be used on chunk that density slicing selected by that | 360 This is intended to be used on chunk that density slicing selected by that |