mercurial-scm/hg: hgext/git/index.py annotate

annotate hgext/git/index.py @ 52627:4dadaf300fe0

git: index changed files on-demand Instead of indexing the changed files for every commit immediately, we can index... 1. heads' changed files immediately 2. other commits' changed files on-demand This helps a lot on repositories with large histories since the initial mercurial invocation doesn't have to wait for the complete repo history to be indexed.

author	Josef 'Jeff' Sipek <jeffpc@josefsipek.net>
date	Fri, 04 Oct 2024 10:51:44 -0400
parents	42f00965e50b
children	3865451a5fab

rev	line source
51859 f4733654f144 typing: add `from __future__ import annotations` to most files Matt Harbison <matt_harbison@yahoo.com> parents: 48875 diff changeset	1 from __future__ import annotations
f4733654f144 typing: add `from __future__ import annotations` to most files Matt Harbison <matt_harbison@yahoo.com> parents: 48875 diff changeset	2
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	3 import collections
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	4 import os
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	5 import sqlite3
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	6
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	7 from mercurial.i18n import _
47012 d55b71393907 node: replace nullid and friends with nodeconstants class Joerg Sonnenberger <joerg@bec.de> parents: 46113 diff changeset	8 from mercurial.node import sha1nodeconstants
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	9
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	10 from mercurial import (
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	11 encoding,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	12 error,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	13 pycompat,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	14 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	15
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	16 from . import gitutil
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	17
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	18
44484 ec54b3d2af0b git: don't fail import when pygit2 is not install Martin von Zweigbergk <martinvonz@google.com> parents: 44477 diff changeset	19 pygit2 = gitutil.get_pygit2()
ec54b3d2af0b git: don't fail import when pygit2 is not install Martin von Zweigbergk <martinvonz@google.com> parents: 44477 diff changeset	20
52626 42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	21 _CURRENT_SCHEMA_VERSION = 4
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	22 _SCHEMA = (
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	23 """
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	24 CREATE TABLE refs (
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	25 -- node and name are unique together. There may be more than one name for
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	26 -- a given node, and there may be no name at all for a given node (in the
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	27 -- case of an anonymous hg head).
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	28 node TEXT NOT NULL,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	29 name TEXT
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	30 );
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	31
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	32 -- The "possible heads" of the repository, which we use to figure out
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	33 -- if we need to re-walk the changelog.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	34 CREATE TABLE possible_heads (
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	35 node TEXT NOT NULL
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	36 );
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	37
52622 aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	38 CREATE UNIQUE INDEX possible_heads_idx ON possible_heads(node);
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	39
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	40 -- The topological heads of the changelog, which hg depends on.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	41 CREATE TABLE heads (
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	42 node TEXT NOT NULL
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	43 );
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	44
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	45 -- A total ordering of the changelog
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	46 CREATE TABLE changelog (
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	47 rev INTEGER NOT NULL PRIMARY KEY,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	48 node TEXT NOT NULL,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	49 p1 TEXT,
52624 cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	50 p2 TEXT,
52626 42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	51 synthetic TEXT,
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	52 changedfiles BOOLEAN
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	53 );
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	54
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	55 CREATE UNIQUE INDEX changelog_node_idx ON changelog(node);
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	56 CREATE UNIQUE INDEX changelog_node_rev_idx ON changelog(rev, node);
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	57
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	58 -- Changed files for each commit, which lets us dynamically build
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	59 -- filelogs.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	60 CREATE TABLE changedfiles (
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	61 node TEXT NOT NULL,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	62 filename TEXT NOT NULL,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	63 -- 40 zeroes for deletions
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	64 filenode TEXT NOT NULL,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	65 -- to handle filelog parentage:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	66 p1node TEXT,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	67 p1filenode TEXT,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	68 p2node TEXT,
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	69 p2filenode TEXT
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	70 );
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	71
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	72 CREATE INDEX changedfiles_nodes_idx
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	73 ON changedfiles(node);
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	74
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	75 PRAGMA user_version=%d
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	76 """
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	77 % _CURRENT_SCHEMA_VERSION
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	78 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	79
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	80
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	81 def _createdb(path):
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	82 # print('open db', path)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	83 # import traceback
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	84 # traceback.print_stack()
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	85 db = sqlite3.connect(encoding.strfromlocal(path))
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	86 db.text_factory = bytes
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	87
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	88 res = db.execute('PRAGMA user_version').fetchone()[0]
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	89
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	90 # New database.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	91 if res == 0:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	92 for statement in _SCHEMA.split(';'):
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	93 db.execute(statement.strip())
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	94
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	95 db.commit()
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	96
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	97 elif res == _CURRENT_SCHEMA_VERSION:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	98 pass
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	99
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	100 else:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	101 raise error.Abort(_(b'sqlite database has unrecognized version'))
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	102
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	103 db.execute('PRAGMA journal_mode=WAL')
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	104
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	105 return db
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	106
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	107
44484 ec54b3d2af0b git: don't fail import when pygit2 is not install Martin von Zweigbergk <martinvonz@google.com> parents: 44477 diff changeset	108 _OUR_ORDER = ()
ec54b3d2af0b git: don't fail import when pygit2 is not install Martin von Zweigbergk <martinvonz@google.com> parents: 44477 diff changeset	109 if pygit2:
ec54b3d2af0b git: don't fail import when pygit2 is not install Martin von Zweigbergk <martinvonz@google.com> parents: 44477 diff changeset	110 _OUR_ORDER = (
ec54b3d2af0b git: don't fail import when pygit2 is not install Martin von Zweigbergk <martinvonz@google.com> parents: 44477 diff changeset	111 pygit2.GIT_SORT_TOPOLOGICAL
ec54b3d2af0b git: don't fail import when pygit2 is not install Martin von Zweigbergk <martinvonz@google.com> parents: 44477 diff changeset	112 \| pygit2.GIT_SORT_TIME
ec54b3d2af0b git: don't fail import when pygit2 is not install Martin von Zweigbergk <martinvonz@google.com> parents: 44477 diff changeset	113 \| pygit2.GIT_SORT_REVERSE
ec54b3d2af0b git: don't fail import when pygit2 is not install Martin von Zweigbergk <martinvonz@google.com> parents: 44477 diff changeset	114 )
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	115
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	116 _DIFF_FLAGS = 1 << 21 # GIT_DIFF_FORCE_BINARY, which isn't exposed by pygit2
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	117
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	118
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	119 def _find_nearest_ancestor_introducing_node(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	120 db, gitrepo, file_path, walk_start, filenode
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	121 ):
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	122 """Find the nearest ancestor that introduces a file node.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	123
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	124 Args:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	125 db: a handle to our sqlite database.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	126 gitrepo: A pygit2.Repository instance.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	127 file_path: the path of a file in the repo
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	128 walk_start: a pygit2.Oid that is a commit where we should start walking
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	129 for our nearest ancestor.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	130
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	131 Returns:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	132 A hexlified SHA that is the commit ID of the next-nearest parent.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	133 """
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	134 assert isinstance(file_path, str), 'file_path must be str, got %r' % type(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	135 file_path
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	136 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	137 assert isinstance(filenode, str), 'filenode must be str, got %r' % type(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	138 filenode
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	139 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	140 parent_options = {
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	141 row[0].decode('ascii')
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	142 for row in db.execute(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	143 'SELECT node FROM changedfiles '
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	144 'WHERE filename = ? AND filenode = ?',
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	145 (file_path, filenode),
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	146 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	147 }
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	148 inner_walker = gitrepo.walk(walk_start, _OUR_ORDER)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	149 for w in inner_walker:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	150 if w.id.hex in parent_options:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	151 return w.id.hex
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	152 raise error.ProgrammingError(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	153 'Unable to find introducing commit for %s node %s from %s',
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	154 (file_path, filenode, walk_start),
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	155 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	156
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	157
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	158 def fill_in_filelog(gitrepo, db, startcommit, path, startfilenode):
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	159 """Given a starting commit and path, fill in a filelog's parent pointers.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	160
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	161 Args:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	162 gitrepo: a pygit2.Repository
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	163 db: a handle to our sqlite database
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	164 startcommit: a hexlified node id for the commit to start at
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	165 path: the path of the file whose parent pointers we should fill in.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	166 filenode: the hexlified node id of the file at startcommit
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	167
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	168 TODO: make filenode optional
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	169 """
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	170 assert isinstance(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	171 startcommit, str
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	172 ), 'startcommit must be str, got %r' % type(startcommit)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	173 assert isinstance(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	174 startfilenode, str
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	175 ), 'startfilenode must be str, got %r' % type(startfilenode)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	176 visit = collections.deque([(startcommit, startfilenode)])
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	177 while visit:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	178 cnode, filenode = visit.popleft()
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	179 commit = gitrepo[cnode]
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	180 parents = []
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	181 for parent in commit.parents:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	182 t = parent.tree
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	183 for comp in path.split('/'):
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	184 try:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	185 t = gitrepo[t[comp].id]
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	186 except KeyError:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	187 break
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	188 else:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	189 introducer = _find_nearest_ancestor_introducing_node(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	190 db, gitrepo, path, parent.id, t.id.hex
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	191 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	192 parents.append((introducer, t.id.hex))
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	193 p1node = p1fnode = p2node = p2fnode = gitutil.nullgit
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	194 for par, parfnode in parents:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	195 found = int(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	196 db.execute(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	197 'SELECT COUNT(*) FROM changedfiles WHERE '
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	198 'node = ? AND filename = ? AND filenode = ? AND '
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	199 'p1node NOT NULL',
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	200 (par, path, parfnode),
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	201 ).fetchone()[0]
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	202 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	203 if found == 0:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	204 assert par is not None
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	205 visit.append((par, parfnode))
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	206 if parents:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	207 p1node, p1fnode = parents[0]
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	208 if len(parents) == 2:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	209 p2node, p2fnode = parents[1]
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	210 if len(parents) > 2:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	211 raise error.ProgrammingError(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	212 b"git support can't handle octopus merges"
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	213 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	214 db.execute(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	215 'UPDATE changedfiles SET '
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	216 'p1node = ?, p1filenode = ?, p2node = ?, p2filenode = ? '
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	217 'WHERE node = ? AND filename = ? AND filenode = ?',
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	218 (p1node, p1fnode, p2node, p2fnode, commit.id.hex, path, filenode),
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	219 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	220 db.commit()
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	221
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	222
52626 42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	223 def _index_repo_commit(gitrepo, db, node, commit=False):
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	224 already_done = db.execute(
52627 4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	225 "SELECT changedfiles FROM changelog WHERE node=?", (node,)
52626 42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	226 ).fetchone()[0]
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	227 if already_done:
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	228 return # This commit has already been indexed
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	229
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	230 commit = gitrepo[node]
52623 4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	231 files = {}
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	232 # I think we only need to check p1 for changed files
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	233 # (and therefore linkrevs), because any node that would
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	234 # actually have this commit as a linkrev would be
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	235 # completely new in this rev.
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	236 p1 = commit.parents[0].id.hex if commit.parents else None
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	237 if p1 is not None:
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	238 patchgen = gitrepo.diff(p1, commit.id.hex, flags=_DIFF_FLAGS)
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	239 else:
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	240 patchgen = commit.tree.diff_to_tree(swap=True, flags=_DIFF_FLAGS)
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	241 new_files = (p.delta.new_file for p in patchgen)
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	242 files = {
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	243 nf.path: nf.id.hex
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	244 for nf in new_files
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	245 if nf.id.raw != sha1nodeconstants.nullid
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	246 }
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	247 for p, n in files.items():
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	248 # We intentionally set NULLs for any file parentage
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	249 # information so it'll get demand-computed later. We
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	250 # used to do it right here, and it was _very_ slow.
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	251 db.execute(
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	252 'INSERT INTO changedfiles ('
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	253 'node, filename, filenode, p1node, p1filenode, p2node, '
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	254 'p2filenode) VALUES(?, ?, ?, ?, ?, ?, ?)',
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	255 (commit.id.hex, p, n, None, None, None, None),
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	256 )
52626 42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	257 # Mark the commit as loaded
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	258 db.execute(
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	259 "UPDATE changelog SET changedfiles=TRUE WHERE node=?", (commit.id.hex,)
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	260 )
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	261 if commit:
42f00965e50b git: track which commit's file changes have been indexed Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52625 diff changeset	262 db.commit()
52623 4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	263
4e2ea270ba6a git: move file indexing into a helper Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52622 diff changeset	264
44951 83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	265 def _index_repo(
83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	266 gitrepo,
83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	267 db,
83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	268 logfn=lambda x: None,
83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	269 progress_factory=lambda args, *kwargs: None,
83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	270 ):
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	271 # Identify all references so we can tell the walker to visit all of them.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	272 all_refs = gitrepo.listall_references()
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	273 possible_heads = set()
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	274 prog = progress_factory(b'refs')
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	275 for pos, ref in enumerate(all_refs):
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	276 if prog is not None:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	277 prog.update(pos)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	278 if not (
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	279 ref.startswith('refs/heads/') # local branch
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	280 or ref.startswith('refs/tags/') # tag
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	281 or ref.startswith('refs/remotes/') # remote branch
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	282 or ref.startswith('refs/hg/') # from this extension
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	283 ):
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	284 continue
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	285 try:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	286 start = gitrepo.lookup_reference(ref).peel(pygit2.GIT_OBJ_COMMIT)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	287 except ValueError:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	288 # No commit to be found, so we don't care for hg's purposes.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	289 continue
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	290 possible_heads.add(start.id)
52625 27a0bfe770eb git: minor wording tweak to a comment Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52624 diff changeset	291 # Optimization: if the list of refs hasn't changed, don't
27a0bfe770eb git: minor wording tweak to a comment Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52624 diff changeset	292 # reindex the changelog. This doesn't matter on small
27a0bfe770eb git: minor wording tweak to a comment Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52624 diff changeset	293 # repositories, but on even moderately deep histories (e.g., cpython)
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	294 # this is a very important performance win.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	295 #
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	296 # TODO: we should figure out how to incrementally index history
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	297 # (preferably by detecting rewinds!) so that we don't have to do a
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	298 # full changelog walk every time a new commit is created.
44946 fb2936c5f6dc git: decode node IDs back into Python strings (issue6349) Hollis Blanchard <hollis_blanchard@mentor.com> parents: 44484 diff changeset	299 cache_heads = {
fb2936c5f6dc git: decode node IDs back into Python strings (issue6349) Hollis Blanchard <hollis_blanchard@mentor.com> parents: 44484 diff changeset	300 pycompat.sysstr(x[0])
fb2936c5f6dc git: decode node IDs back into Python strings (issue6349) Hollis Blanchard <hollis_blanchard@mentor.com> parents: 44484 diff changeset	301 for x in db.execute('SELECT node FROM possible_heads')
fb2936c5f6dc git: decode node IDs back into Python strings (issue6349) Hollis Blanchard <hollis_blanchard@mentor.com> parents: 44484 diff changeset	302 }
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	303 walker = None
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	304 cur_cache_heads = {h.hex for h in possible_heads}
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	305 if cur_cache_heads == cache_heads:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	306 return
44951 83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	307 logfn(b'heads mismatch, rebuilding dagcache\n')
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	308 for start in possible_heads:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	309 if walker is None:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	310 walker = gitrepo.walk(start, _OUR_ORDER)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	311 else:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	312 walker.push(start)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	313
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	314 # Empty out the existing changelog. Even for large-ish histories
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	315 # we can do the top-level "walk all the commits" dance very
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	316 # quickly as long as we don't need to figure out the changed files
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	317 # list.
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	318 db.execute('DELETE FROM changelog')
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	319 if prog is not None:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	320 prog.complete()
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	321 prog = progress_factory(b'commits')
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	322 # This walker is sure to visit all the revisions in history, but
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	323 # only once.
52624 cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	324 pos = -1
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	325 for commit in walker:
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	326 if prog is not None:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	327 prog.update(pos)
47051 de26b9a7ec29 git: consistently use str for parents when rebuilding the index database Matt Harbison <matt_harbison@yahoo.com> parents: 46113 diff changeset	328 p1 = p2 = gitutil.nullgit
52624 cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	329 if len(commit.parents) <= 2:
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	330 if commit.parents:
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	331 p1 = commit.parents[0].id.hex
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	332 if len(commit.parents) == 2:
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	333 p2 = commit.parents[1].id.hex
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	334 pos += 1
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	335 db.execute(
52627 4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	336 'INSERT INTO changelog (rev, node, p1, p2, synthetic, changedfiles) VALUES(?, ?, ?, ?, NULL, FALSE)',
52624 cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	337 (pos, commit.id.hex, p1, p2),
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	338 )
52624 cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	339 else:
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	340 parents = list(commit.parents)
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	341
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	342 p1 = parents.pop(0).id.hex
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	343 while parents:
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	344 pos += 1
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	345
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	346 if len(parents) == 1:
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	347 this = commit.id.hex
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	348 synth = None
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	349 else:
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	350 this = "%040x" % pos
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	351 synth = commit.id.hex
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	352
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	353 p2 = parents.pop(0).id.hex
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	354
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	355 db.execute(
52627 4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	356 'INSERT INTO changelog (rev, node, p1, p2, synthetic, changedfiles) VALUES(?, ?, ?, ?, ?, FALSE)',
52624 cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	357 (pos, this, p1, p2, synth),
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	358 )
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	359
cdbfe5e7592e git: handle octopus merges Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52623 diff changeset	360 p1 = this
52627 4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	361 # Determine heads from the list of possible heads.
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	362 db.execute('DELETE FROM heads')
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	363 db.execute('DELETE FROM possible_heads')
52622 aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	364 db.executemany(
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	365 'INSERT INTO possible_heads (node) VALUES(?)',
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	366 [(hid.hex,) for hid in possible_heads],
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	367 )
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	368 db.execute(
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	369 '''
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	370 INSERT INTO heads (node)
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	371 SELECT node FROM possible_heads WHERE
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	372 node NOT IN (
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	373 SELECT DISTINCT possible_heads.node FROM changelog, possible_heads WHERE
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	374 changelog.p1 = possible_heads.node OR
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	375 changelog.p2 = possible_heads.node
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	376 )
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	377 '''
aa5844ade247 git: speed up possible head processing during indexing by ~100x Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 51859 diff changeset	378 )
52627 4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	379 # Mark all commits with already-loaded changefiles info
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	380 db.execute(
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	381 '''
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	382 UPDATE changelog SET changedfiles=TRUE WHERE node IN (
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	383 SELECT DISTINCT node FROM changedfiles
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	384 )
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	385 '''
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	386 )
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	387
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	388 if prog is not None:
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	389 prog.complete()
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	390
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	391 # Index the changed files for head commits
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	392 prog = progress_factory(b'indexing head files')
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	393 heads = [
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	394 row[0].decode('ascii') for row in db.execute("SELECT * FROM heads")
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	395 ]
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	396 for pos, h in enumerate(heads):
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	397 if prog is not None:
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	398 prog.update(pos)
4dadaf300fe0 git: index changed files on-demand Josef 'Jeff' Sipek <jeffpc@josefsipek.net> parents: 52626 diff changeset	399 _index_repo_commit(gitrepo, db, h)
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	400
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	401 db.commit()
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	402 if prog is not None:
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	403 prog.complete()
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	404
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	405
44951 83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	406 def get_index(
83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	407 gitrepo, logfn=lambda x: None, progress_factory=lambda args, *kwargs: None
83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	408 ):
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	409 cachepath = os.path.join(
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	410 pycompat.fsencode(gitrepo.path), b'..', b'.hg', b'cache'
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	411 )
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	412 if not os.path.exists(cachepath):
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	413 os.makedirs(cachepath)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	414 dbpath = os.path.join(cachepath, b'git-commits.sqlite')
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	415 db = _createdb(dbpath)
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	416 # TODO check against gitrepo heads before doing a full index
ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	417 # TODO thread a ui.progress call into this layer
44951 83e41b73d115 git: add debug logging when there's a mismatch in the cached heads list Augie Fackler <augie@google.com> parents: 44946 diff changeset	418 _index_repo(gitrepo, db, logfn, progress_factory)
44477 ad718271a9eb git: skeleton of a new extension to _directly_ operate on git repos Augie Fackler <augie@google.com> parents: diff changeset	419 return db

Mercurial > public > mercurial-scm > hg

annotate hgext/git/index.py @ 52627:4dadaf300fe0