mercurial-scm/hg-stable: mercurial/worker.py annotate

annotate mercurial/worker.py @ 30421:47de34f79f93

worker: wait worker pid explicitly Before this patch, waitforworkers uses os.wait() to collect child workers, and only wait len(pids) processes. This can have serious issues if other code spawns new processes and does not reap them: 1. worker.py may get wrong exit code and kill innocent workers. 2. worker.py may continue without waiting for all workers to complete. This patch fixes the issue by using waitpid to wait worker pid explicitly. However, this patch introduces a new issue: worker failure may not be handled immediately. The issue will be addressed in next patches.

author	Jun Wu <quark@fb.com>
date	Thu, 28 Jul 2016 20:51:20 +0100
parents	7a5d6e2fd2d5
children	7bc25549e084

rev	line source
18635 fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	1 # worker.py - master-slave parallelism support
fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	2 #
fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	3 # Copyright 2013 Facebook, Inc.
fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	4 #
fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	7
25992 2d76f8a2d831 worker: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	8 from __future__ import absolute_import
2d76f8a2d831 worker: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	9
2d76f8a2d831 worker: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	10 import errno
2d76f8a2d831 worker: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	11 import os
2d76f8a2d831 worker: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	12 import signal
2d76f8a2d831 worker: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	13 import sys
2d76f8a2d831 worker: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	14 import threading
2d76f8a2d831 worker: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	15
2d76f8a2d831 worker: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 25660 diff changeset	16 from .i18n import _
30406 78a58dcf8853 worker: migrate to util.iterfile Jun Wu <quark@fb.com> parents: 28292 diff changeset	17 from . import (
78a58dcf8853 worker: migrate to util.iterfile Jun Wu <quark@fb.com> parents: 28292 diff changeset	18 error,
78a58dcf8853 worker: migrate to util.iterfile Jun Wu <quark@fb.com> parents: 28292 diff changeset	19 util,
78a58dcf8853 worker: migrate to util.iterfile Jun Wu <quark@fb.com> parents: 28292 diff changeset	20 )
18635 fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	21
fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	22 def countcpus():
fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	23 '''try to count the number of CPUs on the system'''
26568 c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	24
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	25 # posix
18635 fed06dd07665 worker: count the number of CPUs Bryan O'Sullivan <bryano@fb.com> parents: diff changeset	26 try:
26568 c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	27 n = int(os.sysconf('SC_NPROCESSORS_ONLN'))
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	28 if n > 0:
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	29 return n
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	30 except (AttributeError, ValueError):
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	31 pass
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	32
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	33 # windows
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	34 try:
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	35 n = int(os.environ['NUMBER_OF_PROCESSORS'])
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	36 if n > 0:
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	37 return n
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	38 except (KeyError, ValueError):
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	39 pass
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	40
c0501c26b05c worker: restore old countcpus code (issue4869) Gregory Szorc <gregory.szorc@gmail.com> parents: 26063 diff changeset	41 return 1
18636 dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	42
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	43 def _numworkers(ui):
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	44 s = ui.config('worker', 'numcpus')
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	45 if s:
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	46 try:
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	47 n = int(s)
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	48 if n >= 1:
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	49 return n
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	50 except ValueError:
26587 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 26568 diff changeset	51 raise error.Abort(_('number of cpus must be an integer'))
18636 dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	52 return min(max(countcpus(), 4), 32)
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	53
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	54 if os.name == 'posix':
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	55 _startupcost = 0.01
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	56 else:
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	57 _startupcost = 1e30
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	58
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	59 def worthwhile(ui, costperop, nops):
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	60 '''try to determine whether the benefit of multiple processes can
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	61 outweigh the cost of starting them'''
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	62 linear = costperop * nops
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	63 workers = _numworkers(ui)
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	64 benefit = linear - (_startupcost * workers + linear / workers)
dcb27c153a40 worker: estimate whether it's worth running a task in parallel Bryan O'Sullivan <bryano@fb.com> parents: 18635 diff changeset	65 return benefit >= 0.15
18637 ac4dbceeb14a worker: partition a list (of tasks) into equal-sized chunks Bryan O'Sullivan <bryano@fb.com> parents: 18636 diff changeset	66
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	67 def worker(ui, costperarg, func, staticargs, args):
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	68 '''run a function, possibly in parallel in multiple worker
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	69 processes.
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	70
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	71 returns a progress iterator
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	72
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	73 costperarg - cost of a single task
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	74
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	75 func - function to run
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	76
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	77 staticargs - arguments to pass to every invocation of the function
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	78
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	79 args - arguments to split into chunks, to pass to individual
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	80 workers
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	81 '''
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	82 if worthwhile(ui, costperarg, len(args)):
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	83 return _platformworker(ui, func, staticargs, args)
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	84 return func(*staticargs + (args,))
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	85
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	86 def _posixworker(ui, func, staticargs, args):
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	87 rfd, wfd = os.pipe()
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	88 workers = _numworkers(ui)
18708 86524a70c0f6 worker: fix a race in SIGINT handling Bryan O'Sullivan <bryano@fb.com> parents: 18707 diff changeset	89 oldhandler = signal.getsignal(signal.SIGINT)
86524a70c0f6 worker: fix a race in SIGINT handling Bryan O'Sullivan <bryano@fb.com> parents: 18707 diff changeset	90 signal.signal(signal.SIGINT, signal.SIG_IGN)
18709 9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	91 pids, problem = [], [0]
30420 7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	92 def killworkers():
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	93 # if one worker bails, there's no good reason to wait for the rest
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	94 for p in pids:
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	95 try:
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	96 os.kill(p, signal.SIGTERM)
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	97 except OSError as err:
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	98 if err.errno != errno.ESRCH:
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	99 raise
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	100 def waitforworkers():
30421 47de34f79f93 worker: wait worker pid explicitly Jun Wu <quark@fb.com> parents: 30420 diff changeset	101 for pid in pids:
47de34f79f93 worker: wait worker pid explicitly Jun Wu <quark@fb.com> parents: 30420 diff changeset	102 st = _exitstatus(os.waitpid(pid, 0)[1])
30420 7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	103 if st and not problem[0]:
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	104 problem[0] = st
7a5d6e2fd2d5 worker: move killworkers and waitforworkers up Jun Wu <quark@fb.com> parents: 30406 diff changeset	105 killworkers()
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	106 for pargs in partition(args, workers):
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	107 pid = os.fork()
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	108 if pid == 0:
18708 86524a70c0f6 worker: fix a race in SIGINT handling Bryan O'Sullivan <bryano@fb.com> parents: 18707 diff changeset	109 signal.signal(signal.SIGINT, oldhandler)
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	110 try:
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	111 os.close(rfd)
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	112 for i, item in func(*(staticargs + (pargs,))):
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	113 os.write(wfd, '%d %s\n' % (i, item))
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	114 os._exit(0)
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	115 except KeyboardInterrupt:
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	116 os._exit(255)
19408 c7ec39c1a381 worker: properly report errors from worker processes (issue3982) Matt Mackall <mpm@selenic.com> parents: 19406 diff changeset	117 # other exceptions are allowed to propagate, we rely
c7ec39c1a381 worker: properly report errors from worker processes (issue3982) Matt Mackall <mpm@selenic.com> parents: 19406 diff changeset	118 # on lock.py's pid checks to avoid release callbacks
18709 9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	119 pids.append(pid)
9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	120 pids.reverse()
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	121 os.close(wfd)
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	122 fp = os.fdopen(rfd, 'rb', 0)
18709 9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	123 t = threading.Thread(target=waitforworkers)
9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	124 t.start()
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	125 def cleanup():
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	126 signal.signal(signal.SIGINT, oldhandler)
18709 9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	127 t.join()
9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	128 status = problem[0]
9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	129 if status:
9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	130 if status < 0:
9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	131 os.kill(os.getpid(), -status)
9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	132 sys.exit(status)
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	133 try:
30406 78a58dcf8853 worker: migrate to util.iterfile Jun Wu <quark@fb.com> parents: 28292 diff changeset	134 for line in util.iterfile(fp):
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	135 l = line.split(' ', 1)
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	136 yield int(l[0]), l[1][:-1]
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	137 except: # re-raises
18709 9955fc5ee24b worker: handle worker failures more aggressively Bryan O'Sullivan <bryano@fb.com> parents: 18708 diff changeset	138 killworkers()
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	139 cleanup()
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	140 raise
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	141 cleanup()
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	142
18707 d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	143 def _posixexitstatus(code):
d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	144 '''convert a posix exit status into the same form returned by
d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	145 os.spawnv
d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	146
d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	147 returns None if the process was stopped instead of exiting'''
d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	148 if os.WIFEXITED(code):
d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	149 return os.WEXITSTATUS(code)
d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	150 elif os.WIFSIGNALED(code):
d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	151 return -os.WTERMSIG(code)
d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	152
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	153 if os.name != 'nt':
047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	154 _platformworker = _posixworker
18707 d1a2b086d058 worker: on error, exit similarly to the first failing worker Bryan O'Sullivan <bryano@fb.com> parents: 18638 diff changeset	155 _exitstatus = _posixexitstatus
18638 047110c0e2a8 worker: allow a function to be run in multiple worker processes Bryan O'Sullivan <bryano@fb.com> parents: 18637 diff changeset	156
18637 ac4dbceeb14a worker: partition a list (of tasks) into equal-sized chunks Bryan O'Sullivan <bryano@fb.com> parents: 18636 diff changeset	157 def partition(lst, nslices):
28181 f8efc8a3a991 worker: change partition strategy to every Nth element Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	158 '''partition a list into N slices of roughly equal size
f8efc8a3a991 worker: change partition strategy to every Nth element Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	159
f8efc8a3a991 worker: change partition strategy to every Nth element Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	160 The current strategy takes every Nth element from the input. If
f8efc8a3a991 worker: change partition strategy to every Nth element Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	161 we ever write workers that need to preserve grouping in input
f8efc8a3a991 worker: change partition strategy to every Nth element Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	162 we should consider allowing callers to specify a partition strategy.
28292 3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	163
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	164 mpm is not a fan of this partitioning strategy when files are involved.
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	165 In his words:
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	166
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	167 Single-threaded Mercurial makes a point of creating and visiting
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	168 files in a fixed order (alphabetical). When creating files in order,
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	169 a typical filesystem is likely to allocate them on nearby regions on
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	170 disk. Thus, when revisiting in the same order, locality is maximized
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	171 and various forms of OS and disk-level caching and read-ahead get a
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	172 chance to work.
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	173
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	174 This effect can be quite significant on spinning disks. I discovered it
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	175 circa Mercurial v0.4 when revlogs were named by hashes of filenames.
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	176 Tarring a repo and copying it to another disk effectively randomized
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	177 the revlog ordering on disk by sorting the revlogs by hash and suddenly
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	178 performance of my kernel checkout benchmark dropped by ~10x because the
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	179 "working set" of sectors visited no longer fit in the drive's cache and
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	180 the workload switched from streaming to random I/O.
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	181
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	182 What we should really be doing is have workers read filenames from a
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	183 ordered queue. This preserves locality and also keeps any worker from
3eb7faf6d958 worker: document poor partitioning scheme impact Gregory Szorc <gregory.szorc@gmail.com> parents: 28181 diff changeset	184 getting more than one file out of balance.
28181 f8efc8a3a991 worker: change partition strategy to every Nth element Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	185 '''
f8efc8a3a991 worker: change partition strategy to every Nth element Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	186 for i in range(nslices):
f8efc8a3a991 worker: change partition strategy to every Nth element Gregory Szorc <gregory.szorc@gmail.com> parents: 26587 diff changeset	187 yield lst[i::nslices]

Mercurial > public > mercurial-scm > hg-stable

annotate mercurial/worker.py @ 30421:47de34f79f93