Mercurial > public > mercurial-scm > hg
comparison mercurial/localrepo.py @ 33389:7e89bd0cfb86
localrepo: cache types for filtered repos (issue5043)
Python introduces a reference cycle on dynamically created types
via __mro__, making them very easy to leak. See
https://bugs.python.org/issue17950.
Previously, repo.filtered() created a type on every invocation.
Long-running processes (like `hg convert`) could call this
function thousands of times, leading to a steady memory leak.
Since we're Unable to stop the leak because this is a bug in
Python, the next best thing is to contain it.
This patch adds a cache of of the dynamically generated repoview/filter
types on the localrepo object. Since we only generate each type
once, we cap the amount of memory that can leak to something
reasonable.
After this change, `hg convert` no longer leaks memory on every
revision. The process will likely grow memory usage over time due
to e.g. larger manifests. But there are no leaks.
| author | Gregory Szorc <gregory.szorc@gmail.com> |
|---|---|
| date | Sat, 01 Jul 2017 20:51:19 -0700 |
| parents | b107a7660f4e |
| children | 1bb209d08a34 |
comparison
equal
deleted
inserted
replaced
| 33388:0823f0983eaa | 33389:7e89bd0cfb86 |
|---|---|
| 428 self.filteredrevcache = {} | 428 self.filteredrevcache = {} |
| 429 | 429 |
| 430 # post-dirstate-status hooks | 430 # post-dirstate-status hooks |
| 431 self._postdsstatus = [] | 431 self._postdsstatus = [] |
| 432 | 432 |
| 433 # Cache of types representing filtered repos. | |
| 434 self._filteredrepotypes = weakref.WeakKeyDictionary() | |
| 435 | |
| 433 # generic mapping between names and nodes | 436 # generic mapping between names and nodes |
| 434 self.names = namespaces.namespaces() | 437 self.names = namespaces.namespaces() |
| 435 | 438 |
| 436 # Key to signature value. | 439 # Key to signature value. |
| 437 self._sparsesignaturecache = {} | 440 self._sparsesignaturecache = {} |
| 537 Intended to be overwritten by filtered repo.""" | 540 Intended to be overwritten by filtered repo.""" |
| 538 return self | 541 return self |
| 539 | 542 |
| 540 def filtered(self, name): | 543 def filtered(self, name): |
| 541 """Return a filtered version of a repository""" | 544 """Return a filtered version of a repository""" |
| 542 # build a new class with the mixin and the current class | 545 # Python <3.4 easily leaks types via __mro__. See |
| 543 # (possibly subclass of the repo) | 546 # https://bugs.python.org/issue17950. We cache dynamically |
| 544 class filteredrepo(repoview.repoview, self.unfiltered().__class__): | 547 # created types so this method doesn't leak on every |
| 545 pass | 548 # invocation. |
| 546 return filteredrepo(self, name) | 549 |
| 550 key = self.unfiltered().__class__ | |
| 551 if key not in self._filteredrepotypes: | |
| 552 # Build a new type with the repoview mixin and the base | |
| 553 # class of this repo. Give it a name containing the | |
| 554 # filter name to aid debugging. | |
| 555 bases = (repoview.repoview, key) | |
| 556 cls = type('%sfilteredrepo' % name, bases, {}) | |
| 557 self._filteredrepotypes[key] = cls | |
| 558 | |
| 559 return self._filteredrepotypes[key](self, name) | |
| 547 | 560 |
| 548 @repofilecache('bookmarks', 'bookmarks.current') | 561 @repofilecache('bookmarks', 'bookmarks.current') |
| 549 def _bookmarks(self): | 562 def _bookmarks(self): |
| 550 return bookmarks.bmstore(self) | 563 return bookmarks.bmstore(self) |
| 551 | 564 |