Cache bug: cached patches truncated to nearest 1024-byte boundary

Cache bug: cached patches truncated to nearest 1024-byte boundary

[mricon]

Hi, all:

It looks like cached patches are truncated to the nearest 1024-byte
boundary in the patch body. E.g.:

> mricon at nikko:[/tmp]$ wget -O no-cache "http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/patch/?id=6e1b4fdad5157bb9e88777d525704aba24389bee"
...
> 2014-06-11 15:34:51 (80.4 MB/s) - ?no-cache? saved [4767]

Patch is complete, without truncation. Next hit, with cache in place:

> mricon at nikko:[/tmp]$ wget -O yes-cache "http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/patch/?id=6e1b4
> fdad5157bb9e88777d525704aba24389bee"
...
> 2014-06-11 15:35:01 (17.0 MB/s) - ?yes-cache? saved [4096/4096]

Length truncated to 4096. The cache on disk looks truncated as well, so
the bug must me during the process of saving cache. The same is true for
larger patches:

> mricon at nikko:[/tmp]$ wget -O no-cache "http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/patch/?id=2840c566e95599cd60c7143762ca8b49d9395050"
...
> 2014-06-11 15:41:33 (1.07 MB/s) - ?no-cache? saved [979644]

979644 bytes with a cache-miss

> mricon at nikko:[/tmp]$ wget -O yes-cache "http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/patch/?id=2840c
> 566e95599cd60c7143762ca8b49d9395050"
...
> 2014-06-11 15:41:46 (1.05 MB/s) - ?yes-cache? saved [978944]

978944 (956KB exactly) with a cache-hit

I've verified this on our other installations and it would appear to
affect them all, not just git.kernel.org systems. Only patches appear to
be truncated -- all other cache files look sane.

This is causing us some grief, so a quickfix would be greatly appreciated!

Best,
-- 
Konstantin Ryabitsev
Senior Systems Administrator
Linux Foundation Collab Projects
Montr?al, Qu?bec

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 538 bytes
Desc: OpenPGP digital signature
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140611/661a816d/attachment.asc>

[PATCH] ui-patch: Flush stdout after outputting data

[john]

Since the "html" functions use raw write(2) to STDIO_FILENO, we don't
notice problems with most pages, but raw patches write using printf(3).
This is fine if we're outputting straight to stdout since the buffers
are flushed on exit, but we close the cache output before this, so the
cached output ends up being truncated.

Make sure the buffers are flushed when we finish outputting a patch so
that we avoid this.

No other UIs use printf(3) so we do not need to worry about them.

Signed-off-by: John Keeping <john at keeping.me.uk>
---
 ui-patch.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ui-patch.c b/ui-patch.c
index 6878a46..fc6c145 100644
--- a/ui-patch.c
+++ b/ui-patch.c
@@ -82,4 +82,6 @@ void cgit_print_patch(const char *new_rev, const char *old_rev,
 		log_tree_commit(&rev, commit);
 		printf("-- \ncgit %s\n\n", cgit_version);
 	}
+
+	fflush(stdout);
 }
-- 
2.0.0.rc4.417.gc642ced

[PATCH] ui-patch: Flush stdout after outputting data

[mricon]

On 11/06/14 04:01 PM, John Keeping wrote:
> Since the "html" functions use raw write(2) to STDIO_FILENO, we don't
> notice problems with most pages, but raw patches write using printf(3).
> This is fine if we're outputting straight to stdout since the buffers
> are flushed on exit, but we close the cache output before this, so the
> cached output ends up being truncated.
> 
> Make sure the buffers are flushed when we finish outputting a patch so
> that we avoid this.
> 
> No other UIs use printf(3) so we do not need to worry about them.

Confirming fix -- in production at kernel.org.
Thanks a ton for the lightning-speed turnaround.

Best,
-- 
Konstantin Ryabitsev
Senior Systems Administrator
Linux Foundation Collab Projects
Montr?al, Qu?bec

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 538 bytes
Desc: OpenPGP digital signature
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140611/cb95e004/attachment.asc>

[PATCH] ui-patch: Flush stdout after outputting data

[john]

On Wed, Jun 11, 2014 at 04:28:26PM -0400, Konstantin Ryabitsev wrote:
> On 11/06/14 04:01 PM, John Keeping wrote:
> > Since the "html" functions use raw write(2) to STDIO_FILENO, we don't
> > notice problems with most pages, but raw patches write using printf(3).
> > This is fine if we're outputting straight to stdout since the buffers
> > are flushed on exit, but we close the cache output before this, so the
> > cached output ends up being truncated.

Actually, it's slightly more interesting than this... since we don't set
GIT_FLUSH, Git decides whether or not it will flush stdout after writing
each commit based on whether or not stdout points to a regular file (in
maybe_flush_or_die()).

Which means that when writing directly to the webserver, Git flushes
stdout for us, but when we redirect stdout to the cache it points to a
regular file so Git no longer flushes the output for us.

The patch is still correct, but perhaps the full explanation is
interesting!

[PATCH] ui-patch: Flush stdout after outputting data

[Jason]

Thanks for patching this John. Merged here:
http://git.zx2c4.com/cgit/commit/?id=2efb59ed0fa8eced79fa702bc47454d3406c3431
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140628/fd940efe/attachment.html>