src/rhodecode: pylons_app/lib/indexers/daemon.py annotate

author	Marcin Kuzminski <marcin@python-works.com>
date	Tue, 17 Aug 2010 23:15:36 +0200
parents
children	0c9dfae57107

rev	line source
439 b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	1 #!/usr/bin/env python
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	2 # encoding: utf-8
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	3 # whoosh indexer daemon for hg-app
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	4 # Copyright (C) 2009-2010 Marcin Kuzminski <marcin@python-works.com>
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	5 #
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	6 # This program is free software; you can redistribute it and/or
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	7 # modify it under the terms of the GNU General Public License
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	8 # as published by the Free Software Foundation; version 2
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	9 # of the License or (at your opinion) any later version of the license.
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	10 #
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	11 # This program is distributed in the hope that it will be useful,
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	14 # GNU General Public License for more details.
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	15 #
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	16 # You should have received a copy of the GNU General Public License
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	17 # along with this program; if not, write to the Free Software
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	18 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	19 # MA 02110-1301, USA.
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	20 """
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	21 Created on Jan 26, 2010
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	22
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	23 @author: marcink
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	24 A deamon will read from task table and run tasks
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	25 """
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	26 import sys
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	27 import os
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	28 from pidlock import LockHeld, DaemonLock
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	29 import traceback
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	30
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	31 from os.path import dirname as dn
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	32 from os.path import join as jn
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	33
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	34 #to get the pylons_app import
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	35 sys.path.append(dn(dn(dn(os.path.realpath(__file__)))))
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	36
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	37 from pylons_app.config.environment import load_environment
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	38 from pylons_app.model.hg_model import HgModel
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	39 from whoosh.index import create_in, open_dir
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	40 from shutil import rmtree
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	41 from pylons_app.lib.indexers import ANALYZER, EXCLUDE_EXTENSIONS, IDX_LOCATION, SCHEMA, IDX_NAME
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	42 import logging
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	43 log = logging.getLogger(__name__)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	44
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	45
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	46 location = '/home/marcink/python_workspace_dirty/*'
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	47
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	48 def scan_paths(root_location):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	49 return HgModel.repo_scan('/', root_location, None, True)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	50
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	51 class WhooshIndexingDaemon(object):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	52 """Deamon for atomic jobs"""
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	53
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	54 def __init__(self, indexname='HG_INDEX'):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	55 self.indexname = indexname
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	56
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	57
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	58 def get_paths(self, root_dir):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	59 """recursive walk in root dir and return a set of all path in that dir
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	60 excluding files in .hg dir"""
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	61 index_paths_ = set()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	62 for path, dirs, files in os.walk(root_dir):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	63 if path.find('.hg') == -1:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	64 for f in files:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	65 index_paths_.add(jn(path, f))
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	66
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	67 return index_paths_
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	68
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	69 def add_doc(self, writer, path, repo):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	70 """Adding doc to writer"""
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	71
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	72 #we don't won't to read excluded file extensions just index them
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	73 if path.split('/')[-1].split('.')[-1].lower() not in EXCLUDE_EXTENSIONS:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	74 fobj = open(path, 'rb')
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	75 content = fobj.read()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	76 fobj.close()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	77 try:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	78 u_content = unicode(content)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	79 except UnicodeDecodeError:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	80 #incase we have a decode error just represent as byte string
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	81 u_content = unicode(str(content).encode('string_escape'))
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	82 else:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	83 u_content = u''
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	84 writer.add_document(owner=unicode(repo.contact),
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	85 repository=u"%s" % repo.name,
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	86 path=u"%s" % path,
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	87 content=u_content,
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	88 modtime=os.path.getmtime(path))
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	89
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	90 def build_index(self):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	91 if os.path.exists(IDX_LOCATION):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	92 rmtree(IDX_LOCATION)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	93
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	94 if not os.path.exists(IDX_LOCATION):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	95 os.mkdir(IDX_LOCATION)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	96
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	97 idx = create_in(IDX_LOCATION, SCHEMA, indexname=IDX_NAME)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	98 writer = idx.writer()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	99
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	100 for cnt, repo in enumerate(scan_paths(location).values()):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	101 log.debug('building index @ %s' % repo.path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	102
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	103 for idx_path in self.get_paths(repo.path):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	104 log.debug(' >> %s' % idx_path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	105 self.add_doc(writer, idx_path, repo)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	106 writer.commit(merge=True)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	107
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	108 log.debug('>>> FINISHED BUILDING INDEX <<<')
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	109
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	110
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	111 def update_index(self):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	112 log.debug('STARTING INCREMENTAL INDEXING UPDATE')
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	113
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	114 idx = open_dir(IDX_LOCATION, indexname=self.indexname)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	115 # The set of all paths in the index
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	116 indexed_paths = set()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	117 # The set of all paths we need to re-index
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	118 to_index = set()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	119
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	120 reader = idx.reader()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	121 writer = idx.writer()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	122
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	123 # Loop over the stored fields in the index
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	124 for fields in reader.all_stored_fields():
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	125 indexed_path = fields['path']
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	126 indexed_paths.add(indexed_path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	127
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	128 if not os.path.exists(indexed_path):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	129 # This file was deleted since it was indexed
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	130 log.debug('removing from index %s' % indexed_path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	131 writer.delete_by_term('path', indexed_path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	132
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	133 else:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	134 # Check if this file was changed since it
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	135 # was indexed
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	136 indexed_time = fields['modtime']
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	137
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	138 mtime = os.path.getmtime(indexed_path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	139
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	140 if mtime > indexed_time:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	141
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	142 # The file has changed, delete it and add it to the list of
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	143 # files to reindex
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	144 log.debug('adding to reindex list %s' % indexed_path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	145 writer.delete_by_term('path', indexed_path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	146 to_index.add(indexed_path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	147 #writer.commit()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	148
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	149 # Loop over the files in the filesystem
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	150 # Assume we have a function that gathers the filenames of the
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	151 # documents to be indexed
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	152 for repo in scan_paths(location).values():
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	153 for path in self.get_paths(repo.path):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	154 if path in to_index or path not in indexed_paths:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	155 # This is either a file that's changed, or a new file
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	156 # that wasn't indexed before. So index it!
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	157 self.add_doc(writer, path, repo)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	158 log.debug('reindexing %s' % path)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	159
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	160 writer.commit(merge=True)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	161 #idx.optimize()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	162 log.debug('>>> FINISHED <<<')
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	163
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	164 def run(self, full_index=False):
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	165 """Run daemon"""
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	166 if full_index:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	167 self.build_index()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	168 else:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	169 self.update_index()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	170
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	171 if __name__ == "__main__":
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	172
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	173 #config = load_environment()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	174 #print config
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	175 try:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	176 l = DaemonLock()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	177 WhooshIndexingDaemon().run(full_index=True)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	178 l.release()
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	179 except LockHeld:
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	180 sys.exit(1)
b153a51b1d3b Implemented search using whoosh. Still as experimental option. Marcin Kuzminski <marcin@python-works.com> parents: diff changeset	181

439

b153a51b1d3b Implemented search using whoosh. Still as experimental option.