comparison mercurial/merge.py @ 28200:588695ccbb22

merge: perform background file closing in batchget As 2fdbf22a1b63 demonstrated with stream clones, closing files on background threads on Windows can yield a significant speedup because closing files that have been created/appended to is slow on Windows/NTFS. Working directory updates can write thousands of files. Therefore it is susceptible to excessive slowness on Windows due to slow file closes. This patch enables background file closing when performing working directory file writes. The impact when performing an `hg up tip` on mozilla-central (136,357 files) from an empty working directory is significant: Before: 535s (8:55) After: 133s (2:13) Delta: -402s (6:42) That's a 4x speedup! By comparison, that same machine can perform the same operation in ~15s on Linux. So Windows went from ~35x to ~9x slower. Not bad but there's still work to do. As a reminder, background file closing is only activated on Windows because it is only beneficial on that platform. So this patch shouldn't change non-Windows behavior at all. It's worth noting that non-Windows systems perform working directory updates with multiple processes. Unfortunately, worker.py doesn't yet support Windows. So, there is still plenty of room for making working directory updates faster on Windows. Even if multiple processes are used on Windows, I believe background file closing will still provide a benefit, as individual processes will still be slowed down by the file close bottleneck (assuming the I/O system isn't saturated).
author Gregory Szorc <gregory.szorc@gmail.com>
date Sat, 20 Feb 2016 15:54:09 -0800
parents d49793aac1ac
children 377f0d8ff874
comparison
equal deleted inserted replaced
28199:d49793aac1ac 28200:588695ccbb22
1056 verbose = repo.ui.verbose 1056 verbose = repo.ui.verbose
1057 fctx = mctx.filectx 1057 fctx = mctx.filectx
1058 wwrite = repo.wwrite 1058 wwrite = repo.wwrite
1059 ui = repo.ui 1059 ui = repo.ui
1060 i = 0 1060 i = 0
1061 if True: 1061 with repo.wvfs.backgroundclosing(ui, expectedcount=len(actions)):
1062 for f, (flags, backup), msg in actions: 1062 for f, (flags, backup), msg in actions:
1063 repo.ui.debug(" %s: %s -> g\n" % (f, msg)) 1063 repo.ui.debug(" %s: %s -> g\n" % (f, msg))
1064 if verbose: 1064 if verbose:
1065 repo.ui.note(_("getting %s\n") % f) 1065 repo.ui.note(_("getting %s\n") % f)
1066 1066
1075 util.rename(absf, orig) 1075 util.rename(absf, orig)
1076 except OSError as e: 1076 except OSError as e:
1077 if e.errno != errno.ENOENT: 1077 if e.errno != errno.ENOENT:
1078 raise 1078 raise
1079 1079
1080 wwrite(f, fctx(f).data(), flags) 1080 wwrite(f, fctx(f).data(), flags, backgroundclose=True)
1081 if i == 100: 1081 if i == 100:
1082 yield i, f 1082 yield i, f
1083 i = 0 1083 i = 0
1084 i += 1 1084 i += 1
1085 if i > 0: 1085 if i > 0: