Mercurial > public > mercurial-scm > hg-stable
diff mercurial/util.py @ 7396:526c40a74bd0
templater: return data in increasing chunk sizes
Currently hgweb is not streaming its output -- it accumulates the
entire response before sending it. This patch restores streaming
behaviour. To avoid having to synchronously write many tiny fragments,
this patch also adds buffering to the template generator. Local
testing of a fetch of a 100,000 line file with wget produces a slight
slowdown overall (up from 6.5 seconds to 7.2 seconds), but instead of
waiting 6 seconds for headers to arrive, output begins immediately.
| author | Brendan Cully <brendan@kublai.com> |
|---|---|
| date | Fri, 21 Nov 2008 15:51:40 -0800 |
| parents | 00d76fa3ffba |
| children | 040484030491 |
line wrap: on
line diff
--- a/mercurial/util.py Sat Nov 22 16:57:49 2008 +0100 +++ b/mercurial/util.py Fri Nov 21 15:51:40 2008 -0800 @@ -290,6 +290,37 @@ l.sort() return l +def increasingchunks(source, min=1024, max=65536): + '''return no less than min bytes per chunk while data remains, + doubling min after each chunk until it reaches max''' + def log2(x): + if not x: + return 0 + i = 0 + while x: + x >>= 1 + i += 1 + return i - 1 + + buf = [] + blen = 0 + for chunk in source: + buf.append(chunk) + blen += len(chunk) + if blen >= min: + if min < max: + min = min << 1 + nmin = 1 << log2(blen) + if nmin > min: + min = nmin + if min > max: + min = max + yield ''.join(buf) + blen = 0 + buf = [] + if buf: + yield ''.join(buf) + class Abort(Exception): """Raised if a command needs to print an error and exit."""